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FOREWORD 


This  report  is  an  outgrowth  of  the  work  of  the 
Defense  Science  Board  Task  Force  on  Test  and  Evaluation, 
and  the  checklists  herein  have  been  derived  from  the 
study  of  past  major  weapon  system  programs. 

The  T&E  expert  in  reading  this  volume  will  find 
many  precepts  which  will  strike  him  as  being  too  obvious 
to  be  included  in  checklists  of  this  type.  These  items 
are  included  because  examples  were  found  where  even  the 
obvious  has  been  neglected,  not  because  of  incompetence 
or  lack  of  personal  dedication  by  the  people  in  charge 
of  the  program,  but  because  of  financial  and  temporal 
pressures  which  forced  competent  managers  to  compromise 
on  their  principles.  It  is  hoped  that  the  inclusion  of 
the  obvious  will  prevent  repetition  of  the  serious  errors 
which  have  been  made  in  the  past  when  such  political, 
economic  and  temporal  pressures  have  forced  project 
managers  to  depart  from  the  rules  of  sound  engineering 
practices. 

In  the  long  run,  taking  short  cuts  during  T&E  to 
save  time  and  money  will  result  in  significant  increases 
in  the  overall  costs  of  the  programs  and  in  the  delay  of 
the  delivery  of  the  corresponding  weapon  systems  to  the 
combatant  forces. 


T&E  GUIDELINES  FOR  AIRCRAFT  SYSTEMS 

The  checklist  items  presented  here  are  specifically  applicable  to 
aircraft  testing  and  evaluation.  It  is  suggested  that  the  user  of  this 
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volume  also  refer  to  the  Report  of  the  Defense  Science  Board  on  Test  and 
Evaluation  which  contains  general  checklist  items  also  applicable  to  this 
system  T&E  program.  The  checklist  items  presented  here  are  organized  into 
time  phases  of  the  acquisition  process  oriented  to  the  DSARC  cycle. 

The  checklists  cover  various  aspects  of  the  major  activities  that 
should  be  underway  during  a  given  time  period.  Hence,  a  checklist  might 
cover  the  (1)  evaluation  of  work  that  occurred  in  the  previous  phase, 

(2)  conduct  of  tests  planned  in  the  previous  phase  and  executed  in  the 
subject  phase,  and  (3)  plans  and  other  preparatory  actions  for  test  sche¬ 
dules  to  be  conducted  in  a  subsequent  phase.  For  reasons  such  as  this, 
items  on  some  subjects,  such  as  development  test  plans,  may  appear  in  more 
than  one  phase.  In  addition,  since  the  Services  and  the  DSARC  have  flexi¬ 
bility  in  deciding  how  rapidly  to  progress  in  the  Validation  Phase,  there 
may  be  cases  where  the  Request  for  Proposals  (RFPs) ,  proposal  evaluations, 
source  selections,  or  contract  negotiations  may  occur  after  the  DSARC 
approves  full-scale  development  instead  of  before.  For  this  reason,  it 
is  recommended  that  previous  checklists  in  the  Validation  Phase  be  re¬ 
viewed  when  entering  the  Full-Scale  Engineering  Development  Phase.  The 
following  are  the  phases  used  in  this  report. 

CONCEPTUAL  PHASE 

The  checklist  items  in  this  phase  are  for  guidance  in  evaluating  T&E 
activities  during  the  Conceptual  Phase  of  the  acquisition  of  the  system. 
This  phase  (often  research  and  exploratory  development)  precedes  the  first 
DSARC  milestone  and  is  focused  on  the  development  of  a  weapon  system  con¬ 
cept  that  offers  high  prospects  of  satisfying  an  identified  military  need. 

Although  not  called  for  in  DoD  Directive  5000.1  specifically,  the 
objectives  of  this  phase  should  be: 
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1.  To  verify  chat  there  is  a  military  need  for  the  proposed 
system. 

2.  To  demonstrate  that  there  is  a  sound  physical  basis  for  a  new 
weapon  system. 

3.  To  formulate  a  concept,  based  on  demonstrated  physical 
phenomena,  for  satisfying  the  military  need. 

4.  To  show  that  the  proposed  solution  is  superior  to  its  com¬ 
petitors  in  terms  of  potential  effectiveness,  probability  of 
success,  probable  cost,  impact  on  the  U.S.  military  posture, 
and  development  risks. 

5.  To  analyze  the  technology  outlook  and  the  military  need  to 
show  that  it  is  better  to  start  advanced  development  now 
rather  than  to  wait  for  future  technological  improvements. 

6.  To  identify  the  key  risk  areas  and  critical  issues  that  need 
to  be  resolved  before  full-scale  development  is  Initiated. 

The  most  important  product  of  this  phase  is  the  Development  Concept 
Paper  (DCP)  or  its  equivalent.  The  DCP  defines  program  Issues,  including 
special  logistics  problems,  program  objectives,  program  plans,  performance 
parameters,  areas  of  major  risk,  system  alternatives,  and  acquisition 
strategy. 


VALIDATION  PHASE 

The  checklist  items  in  this  pha;e  are  for  guidance  in  conducting  T&E 
during  the  Validation  Phase  (the  time  between  when  the  DSARC  recommends 
approval  of  the  DCP  for  the  first  time  and  when  the  DSARC  recommends  full- 
scale  development  of  the  system) . 

'While  these  objectives  are  not  spelled  out  in  the  DoD  Directive  5000.1, 
the  objectives  of  the  Validation  Phase  should  be  to  confirm: 

1.  The  need  for  the  selected  system  in  consideration  of  the  threat, 
system  alternatives,  special  logistics  needs,  estimates  of 
development  costs,  preliminary  estimates  of  life  cycle  costs 
and  potential  benefits  in  context  with  overall  DoD  strategy  and 
fiscal  guidance. 

2.  The  validity  of  the  operational  concept. 

3.  That  development  risks  have  been  identified  and  solutions  are 
in  hand. 

4.  Realism  of  the  plan  for  full-scale  development. 
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In  the  pursuit  of  the  above  objectives,  it  is  likely  that  advanced 
development  T&G  will  be  conducted  to  resolve  issues.  In  some  cases,  an 
RFP  for  full-scale  engineering  development  will  be  prepared,  proposals 
will  be  received  and  evaluated,  and  contracts  negotiated  in  preparation 
for  seeking  DSARC  approval  for  the  next  phase.  Therefore,  some  checklist 
items  are  included  to  help  ensure  that  this  work  properly  reflects  the 
T&E  interests  in  this  and  subsequent  phases.  For  example,  the  RFP  must 
Include  adequate  guidance  to  ensure  that  sufficient  resources  and  time  are 
available  so  that  the  engineering  effort  can  properly  support  the  initial 
DT&E  with  hardware,  software,  technical  data,  and  training. 

The  primary  emphasis  of  OSD/T&E  activities  is  with  items  3  and  4  above. 
Special  attention  should  be  given  to  the  planning  of  IOT&E  activity  as  it 
is  incorporated  in  the  engineering  development  contract  as  well  as  tue 
DT&E  associated  with  addressing  the  critical  issues  and  areas  of  major 
risk  identified  in  the  TCP. 

FULL-SCALE  ENGINEERING  DEVELOPMENT  PHASE 

The  checklist  items  contained  in  this  phase  are  for  guidance  in 
conducting  T&E  during  the  Full-Scale  Engineering  Development  Phase.  This 
Includes  the  major  DT&E  and  the  IOT&E  conducted  prior  to  the  major  pro¬ 
duction  decision.  By  this  time,  the  system  Is  well-defined  and  is  becom¬ 
ing  a  unique  item  and,  hence,  sound  judgment  must  be  applied  in  using  these 
checklist  items. 

To  enter  the  Engineering  Development  Phase,  the  DSARC  will  have: 

•  Confirmed  the  need  in  consideration  of  the  threat,  alternatives, 
logistic  needs,  cost,  and  benefits. 

•  Identified  development  risks. 

•  Confirmed  the  realism  of  the  development  plan. 

Given  the  above,  the  primary  objectives  of  the  DT&E  should  be  to: 

1.  Demonstrate  that  the  engineering  and  design  and  development 
process  is  complete  and  that  the  design  risks  have  been  mini¬ 
mized  (the  system  is  ready  for  production) . 

2.  Demonstrate  that  the  system  will  meet  specifications. 
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The  primary  objectives  of  the  IOT&E  should  be  to: 

3.  Assess  operational  suitability  and  effectiveness. 

4.  Validate  organizational  and  employment  concepts. 

f.  Determine  training  and  logistic  requirements. 

In  addition,  the  validity  of  the  plan  for  the  remainder  of  the  program 
must  be  confirmed  by  the  DSARC  before  substantial  production/deployment 
will  be  recommended  to  the  Secretary  of  Defense. 

The  level  of  OSD/T&E  activity  is  highest  during  this  phase.  The 
IOT&E  plan  must  be  designed,  the  tests  conducted,  and  the  data  analyzed 
to  evaluate  the  inputs  associated  with  the  primary  objectives.  These 
tests  should  not  be  conducted  until  the  primary  objectives  of  the  DT&E 
have  been  met.  Thus,  OSD/T&E  activity  is  required  to  assess  that  the  DT&E 
major  milestone — the  system  is  ready  for  production — has  been  achieved. 
Close  monitoring  of  the  T&E  Service  activity  is  required  during  the  latter 
stages  of  this  phase. 

SUBSTANTIAL  PRODUCT  ION /DEPLOYMENT  PHASE 

The  checklist  items  contained  in  this  phase  are  for  guidance  in  con¬ 
ducting  T&E  after  the  substantial  production  decision  has  been  made  by  the 
DSARC.  This  includes  DT&E  and  follow-on  OT&E  to  be  conducted  on  the  early 
production  items. 

To  enter  the  Production/Deployment  Phase,  the  DSARC  will  have  re¬ 
viewed  the  program  to  confirm: 

•  The  need  for  the  system. 

•  A  practical  engineering  design  with  adequate  consideration  of 
production  and  logistic  problems  is  complete. 

•  All  technical  uncertainties  have  been  resolved  and  opera¬ 
tional  suitability  has  been  determined  by  T&E. 

•  The  realism  of  the  plan. 

The  primary  objective  of  the  DT&E  in  this  phase  should  be  to: 

1.  Verify  that  the  production  system  meets  specifications. 

The  primary  objectives  of  the  follow-on  OT&E  should  be  to: 


4 


2.  Validate  the  operational  suitability  and  effectiveness. 

3.  Optimize  organization  and  doctrine. 

4.  Validate  training  and  logistic  requirements. 

At  this  point,  the  OSD/T&E  activity  is  similar  to  that  in  the 
previous  phase;  however,  much  of  the  testing  is  verification  that  the 
proauction  system  performance  is  as  expected.  Hence,  most  of  the  items 
in  the  previous  phase  are  appropriate  to  this  phase,  especially  those 
related  to  OT&E. 
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I.  CONCEPTUAL  PHASE 


During  this  phase  the  program  is  being  conceived  and  the  DCP  is  being 
prepared.  The  test  and  evaluation  checklist  covers  the  following: 

1.  Test  Program/Total  Costs 

2.  Test  Facilities  and  Instrumentation 

3.  Test  Resources  and  Failures 

4.  System  Interfaces 

5.  Major  Weapon  Subsystems 

6.  Propulsion  System 

7.  0, '.’rational  Scenario 

8.  Evaluation  Criteria 

9.  Untried  Elements 

10.  Brassboard  Avionics  Tests 

11.  Nuclear  Weapons  Effects 
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1.  TEST  PROGRAM/TOTAL  COSTS 

Prior  to  DSARC  I.  all  the  phases  of  the  aircraft  test  program  should 
be  considered  so  that  the  total  costs  and  the  development  schedules 
include  consideration  of  all  likely  activities  in  the  overall  program. 

In  an  aircraft  program  this  might  include  the  cost  of  related  missile 
interface  development,  missiles  used,  new  test  range  instrumentation,  drone 
or  manned  targets,  new  facility  requirements,  chase  aircraft,  the  cost  of 
the  OT&E  program  (both  IOT&E  and  follow-on  OT&E  including  test  gap  closing 
provisions),  the  direct  costs  of  test  support  provided  by  national  test 
facilities  and  the  cost  of  special  tests  such  as  underground  nuclear  radia¬ 
tion  exposure  tests,  EMP  tests,  and  offensive/defensive  tests.  Include 
these  costs  even  if  they  cannot  be  accurately  predicted.  If  they  are  not 
included  now,  they  will  show  up  later  as  cost  overruns. 

2.  TEST  FACILITIES  AND  INSTRUMENTATION 

Before  DSARC  I,  the  test  facilities  and  instrumentation  requirements 
to  conduct  tests  should  be  generally  identified  along  with  a  tentative 
schedule  of  test  activities. 

The  applicability  of  the  test  ranges,  especially  for  attack  aircraft, 
and  the  adequacy  of  the  facilities  and  instrumentation  should  be  verified; 
insofar  as  possible  alternative  approaches  (different  ranges,  etc.)  and 
instrumentation  improvements  needed  should  be  specified.  Funding  for  direct 
costs  of  range  and  instrumentation  s'-nport  must  also  be  identified.  Of 
prime  importance  are  the  constraints  to  be  placed  on  the  test  because  of  the 
range  and  instrumentation.  Factors  .<  \ch  as  available  air-space,  realism  of 
the  ground  targets  and  the  terrain  surrounding  the  area  of  the  instrumented 
range,  etc.  should  not  constrain  the  realism  of  testing.  If  range  and  in¬ 
strumentation  factors  are  found  to  cast  significant  doubt  on  the  meaningful¬ 
ness  of  the  test  data  because  of  a  lack  of  operational  realism,  the  steps 
necessary  to  assure  meaningful  data  should  be  identified  and  planned  before 
the  plans  are  included  in  the  test  concept. 

Targets  for  air-to-air  fighter  aircraft  warrant  specific  mention;  some 
of  the  aircraft  subsystems  cannot  be  adequately  tested  without  targets  that 
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realistically  simulate  the  threat.  In  some  cases,  considerable  lead  time 
is  required  to  provide  the  required  target  support;  hence,  target  require¬ 
ments  to  support  the  tests  must  be  identified  early. 


3.  TEST  RESOURCES  AND  FAILURES 

Ensure  that  there  are  adequate  funds,  reasonable  time  and  an  accept¬ 
able  number  of  aircraft  planned  for  the  various  test  program  phases 
and  that  these  make  provisions  for  the  occurrence  of  failures. 


Some  major  problems  in  test  programs  have  been: 

(a)  Insufficient  flying  hours  not  only  of  the  all  up  aircraft  but 
also  of  subsystems  and  components  to  test  all  modes  that  stress 
the  system  throughout  the  performance  envelope  and  to  develop 
reliability.  Circumstances  have  existed  where,  for  various 
good  reasons,  the  technology  has  had  to  be  pushed  rapidly,  the 
higher  risks  involved  have  required  more  thorough  testing  and/ 
or  greater  resources.  These  typically  have  taken  a  longer 
time  to  complete. 

(b)  Some  significant  problems  found  during  development  have  not 
been  adequately  corrected  prior  to  production.  For  example,  a 
wing  structure  found  to  have  Inadequate  structural  life  as  a 
result  of  R&D  fatigue  tests  should  be  corrected  before  pro¬ 
duction.  Requalification  of  piece  parts  and  assemblies  has 
also  frequently  been  required  for  many  reasons.  This  results 
in  a  need  for  additional  testing. 

(c)  The  schedule  was  based  on  overall  success  and,  among  other 
things,  did  not  provide  for  accidents.  "Plan  tests  assuming 
some  failures".  A  percentage  of  the  total  tests  (sorties, 
runs,  trials,  experiments)  should  be  allowed  for  retesting, 
over  and  above  the  basic  number  of  successful  runs  required. 

There  must  be  sufficient  time  and  resources  put  into  the  early  plan¬ 
ning  for  the  operational  tests,  particularly  the  IOT&E  phase,  because  the 
substantial  production  and  deployment  decision  is  planned  to  be  made  based 
upon  IOT&E  results.  Further,  test  funds  should  not  be  postponed  cr  reori¬ 
ented  to  keep  total  program  costs  in  line.  In  one  program,  it  was  found 
that  testing  time  was  arbitrarily  decreased  by  90  percent  and  the  number 
of  test  aircraft  reduced  to  30  percent  of  the  original  forecast  to  compen¬ 
sate  for  the  effect  of  the  cost  of  development  problems.  One  should  not 
lose  sight  of  the  continuing  need  for  adequate  test  articles.  In  the  past, 
for  example,  there  have  been  cases  of  new  and  unanticipated  weapon  launch 
requirements  which  necessitated  additional  testing. 


4. 


SYSTEM  INTERFACES 


Consider  all  aircraft  system  Interfaces,  their  test  requirements  and 
probable  costs  at  the  outset  of  the  conceptual  phase. 

Ensure  that  the  program  plan  assembled  before  DSARC  1  includes  broad 
test  plans  for  the  entire  program,  and  that  these  include  scheduling  time 
and  costs  required  for  testing  system  interfaces.  The  initial  program 
plans  sometimes  have  not  anticipated  the  need  for  adequate  interface  test¬ 
ing.  For  example,  if  the  system  includes  use  of  air-to-air  or  eir-to- 
surface  missiles,  ensure  that  provisions  are  made  for  testing  them  in 
combination  with  avionics  system  tests.  With  the  advent  of  the  so-called 
smart  weapons  using  E-0,  IR  or  laser  guidance  systems,  the  interface  with 
the  aircraft  is  not  only  on  the  carriage  stations  and  the  fire  control 
system  but  through  the  necessary  displays.  Tests  need  to  be  conducted  to 
ensure  that  the  planned  displays  do  not  limit  the  system  performance.  In 
one  program,  late  selection  of  a  new  avionics  suit  for  the  aircraft  made 
it  impossible  to  define  the  missile/aircraft  interface  at  the  start  of  the 
fixed  price  contracts.  This  led  to  several  design  iterations  to  make  the 
interface  work  correctly. 

5.  MAJOR  WEAPON  SUBSYSTEMS 

If  the  aircraft  system  relies  on  the  successful  development  of  a 
specific  and  separately  funded  major  weapon  (such  as  a  gun  or  missile) 
in  order  to  accomplish  its  primary  mission,  this  major  subsystem 
should  be  developed  and  tested  concurrently  with  or  prior  to  the 
aircraft. 

For  example,  if  the  major  weapon  is  a  gun,  it  should  be  R&D  qualified 
in  ground  tests  and  then  tested  on  the  aircraft  as  early  as  practicable  to 
develop  satisfactory  gun  operation  under  aircraft  G's  and  temperatures,  to 
determine  vibration  effects,  to  investigate  system  integration  problems, 
etc.  Then,  the  aircraft  system  should  be  quickly  subjected  to  a  life-time 
of  gun  fir'ng,  e.g.,  simulated  combat  with  the  reasonable  sorties  and 
rounds  expended. 
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There  have  been  fighter  programs  where  the  gun  tests  were  run  late  and 
problems  were  encountered  with  gun  gas  ingestion  and  link  and  case  ejection 
damage. 

6.  PROPULSION  SYSTEM 

If  the  aircraft  program  is  paced  by  the  propulsion  system  development. 
an  early  advanced  development  project  for  the  propulsion  may  be 
appropriate  for  a  new  concept. 

The  progress  in  the  development  of  the  propulsion  system  is  normally 
a  key  factor  in  determining  when  full-scale  aircraft  development  is  war¬ 
ranted.  However,  when  other  subsystems  such  as  the  avionics,  are  also 
critical  and  when  there  is  a  time  urgency  to  get  the  new  system,  there 
have  been  isolated  examples  where  the  aircraft  system  development  program 
has  been  initiated  with  an  interim  power  plant.  This  requires  additional 
engineering  and  testing  and  can  only  be  justified  in  unusual  cases.  Tests 
should  be  planned  which  use  the  substitute  power  plant  and  should  be  re¬ 
stricted  to  providing  a  platform  for  other  subsystem  flights  tests.  Full 
system  tests  should  be  planned  only  with  the  proposed  power  plant. 

7 .  OPERATIONAL  SCENARIO 

A  conceptual  operational  scenario  for  operation  and  use  of  the  air¬ 
craft  should  be  developed  so  that  general  test  plans  can  be  designed. 
This  should  include  purpose,  roles  and  missions,  threats,  operating 
environments,  logistics  and  maintenance,  and  basing  characteristics. 
The  potential  range  of  values  on  these  aspects  should  be  stated. 

Whenever  typical  scenarios  do  not  exist,  one  can  expect  problems  in 
creating  meaningful  test  plans.  These  should  be  developed  as  soon  as 
practical,  but  in  view  of  the  long  time  interval  between  the  pre-DSARC  I 
period  and  OT&E  it  is  essential  that  T&E  plans  make  provision  for  reviews 
and  changes  of  scenarios.  This  requirement  will  obviously  have  contract¬ 
ual  implications  but  it  is  better  to  anticipate  the  inevitable  necessity 
for  change  than  pretending  to  be  surprised  by  events  that  can  be  foreseen; 
changes  will  always  be  required  in  all  systems  but  especially  in  aircraft 
mission  scenarios.  For  example,  the  mission  for  a  strategic  bomber  became 
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chat  of  a  tactical  support  weapon  for  ground  forces  in  SEA.  Several  air¬ 
craft  developed  during  the  1960s  required  development  and  test  changes  to 
achieve  a  hot  day  performance  for  SEA  operations.  Development  had  been 
initially  pursued  based  on  standard  day  temperatures.  The  aircraft  had  to 
be  modified  to  provide  larger  wings  to  achieve  desired  hot  day  performance. 

8.  EVALUATION  CRITERIA 

Develop  evaluation  criteria  to  be  used  for  the  selection  of  the  final 

aircraft  system  design. 

The  evaluation  criteria  should  be  based  on  performance  factors  which 
are  measurable  through  testing.  For  example,  if  the  aircraft  is  designed 
to  provide  close  air  support  under  adverse  weather  conditions,  these  should 
be  specified;  if  a  helicopter  is  to  be  designed  to  provide  nap  of  the  earth, 
pop-up  and  hover  capability,  these  performance  factors  should  be  specified. 
An  evaluation  plan  should  be  developed  which  describes  the  range  of  accep¬ 
table  performance  for  each  factor.  If  competitive  designs  are  under  con¬ 
sideration,  criteria  for  selection  should  be  specified  in  advance,  with 
critical  issues  identified  for  each  design. 

9.  UNTRIED  ELEMENTS 

The  aircraft  development  program  should  include  conclusive  testing 

to  eliminate  uncertainties  of  the  untried  elements. 

Untried  technology  and  its  application  must  be  backed  by  reasonable 
laboratory  tests.  Untried  concepts  have  been  incorporated  in  aircraft  on 
the  basis  of  paper  studies  and  engineering  analyses,  and  later  Lhe  develop¬ 
ment  or  the  production  item  tests  indicated  that  the  initial  analyses  were 
in  error.  Testing  should  be  planned  to  verify  the  analysis  on  all  critical 
items. 

If  the  aircraft  is  intended  to  incorporate  a  large  number  of  technol¬ 
ogy  improvements  designed  to  provide  operational  improvements  over  current 
systems,  test  milestones  to  demonstrate  that  the  technology  is  in  hand 
should  exist  for  each  of  the  improvements.  Success  should  be  proved  for 
propulsion,  fire  control,  advanced  structural  designs,  etc.  before  the 
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system  is  considered  for  engineering  development.  Be  aware  that,  if  there 
are  a  large  number  of  relatively  modest  new  technology  applications  in  the 
new  system,  it  may  become  difficult  to  conduct  total  system  testing  because 
of  the  high  probability  of  failures  brought  about  by  the  large  number  of 
new  technology  applications. 

All  of  the  untried  elements  of  the  proposed  aircraft  should  be  identi¬ 
fied.  Risks  related  to  these  and  to  alternative  approaches  should  be 
evaluated  and  a  test  program  should  be  conducted.  Untried  items  can  be 
made  low  risk  even  when  failure  potential  is  high  if  good  alternatives 
exist.  Other  untried  items  which  would  appear  technically  to  be  of  low 
risk  can  present  a  major  problem  if  they  later  prove  unworkable,  and  if 
alternatives  do  not  exist.  For  example,  in  one  program  where  the  new  ro¬ 
tor  system  feasibility  had  been  based  on  tests  in  a  smaller  helicopter, 
the  rotor  system  was  considered  as  a  low  technical  risk  and  testing  was 
planned  accordingly.  However,  the  analysis  of  risk  was  wrong,  major  prob¬ 
lems  were  encountered  and  the  program  was  cancelled  after  major  cost  and 
schedule  overruns. 


10.  BRASSBOARD  AVIONICS  TESTS 

The  use  of  brassboard  or  modified  existing  hardware  to  "prove"  that 
the  concept  will  work  should  be  seriously  scrutinized  to  ensure  that 
the  demonstrations  and  tests  are  applicable. 


When  tests  are  conducted  on  aircraft  avionics  that  are  in  the  brass- 
board  stage  or  are  modified  existing  hardware,  the  system  should  be  evalu¬ 
ated  with  special  attention  to  such  as  the  following: 

(a)  Will  the  packaging  of  the  hardware  significantly  affect  the 
performance  characteristics  so  that  the  suggested  proof  of 
validation  is  inconclusive? 

(b)  Will  scaling  laws  invalidate  the  findings  or  introduce  new 
technology  problems?  E.g.,  will  scaling  of  the  radar  affect 
performance? 

(c)  Will  the  laboratory-type  environment  in  which  the  hardware  is 
tested  preclude  the  generation  of  data  needed  to  validate  that 
the  concept  and  technology  approach  will  be  applicable  to  an 
operational  environment?  Has  the  aircraft  environment  (e.g., 
vibrations  from  gun  firing,  sonic  environment  from  engine  noise, 
etc.)  been  taken  into  account? 
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(d)  Do  the  tests  Include  signals,  noise,  and  vibrations  (aerodynamic 
and  gunfire)  sources  representative  of  those  that  might  be  expec¬ 
ted  in  an  operational  environment?  Has  the  likely  EMI  and  ECM 
environment  been  accounted  for? 


11.  NUCLEAR  WEAPONS  EFFECTS 

The  subject  of  nuclear  weapons  effects  should  be  addressed  in  the  test 
concept  for  all  aircraft  weapons  systems  where  operational  suitability 
dictates  that  survivable  exposure  to  nuclear  weapons  effects  is  a 
requirement . 

In  most  aircraft  systems,  inadequate  attention  has  been  given  to  the 
fact  that  they  are  intended  for  possible  operation  in  a  nuclear  environ¬ 
ment.  Following  early  design  for  adequate  nuclear  hardness  of  subsystems 
and  tests  of  the  components  and  subsystems,  at.  "all  up"  aircraft  should  be 
subjected  to  "threat  level"  EMP.  A  "pure"  operational  vehicle  is  required 
for  EMP  tests  since  special  instrumentation  and  subsystem  prototype  pecu¬ 
liarities  might  significantly  bias  EMP  tests.  This  use  of  a  production 
aircraft  must  be  weighed  against  the  economics  of  early  identification  of 
modifications  to  improve  survivability  from  nuclear  weapons  effects.  Early 
planning  must  recognize  that  other  agencies  will  be  involved  (i.e.,  DNA 
and  AEC) . 
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II.  VALIDATION  PHASE 


In  the  Validation  Phase  the  issues  raised  by  the  DCP  may  be  resolved 
by  conducting  tests  on  the  system.  The  development  aiU  operational  plans 
will  be  defined  in  considerable  detail.  The  reader  should  review  the 
Phase  I  checklist  items  since  many  of  them  will  be  appropriate  to  this 
phase.  Additional  test  and  evaluation  checklist  items  are: 

1.  Test  Plans  and  Criteria 

2.  Milestones  and  Goals 

3.  Operating  Concepts  and  Environment 

4.  Priority  of  Requirements 

5.  Test  Program  Building  Blocks 

6.  Technology  Concepts 

7.  DT&E/IOT&E  Plan 

8.  Test  Failures 

9.  Joint  Testing 

10.  Traceability 

11.  Competitive  Prototype  Tests 

12.  Prototype  Similarity  to  Development  and  Production  Aircraft 

13.  Prototype  Tests 

14.  Inlet/Engine/Nozzle  Match 

15.  Subsystem  Tests 

16.  Propulsion  System 

17.  EMI  Testing 

18.  Parts  Interchange 

19.  Human  Factors  Demonstration 

20.  Contract  Form 

21.  Government  Furnished  Equipment 

22.  Military  Preliminary  Evaluation 

23.  User  Participation 

24.  Maintenance  and  Training  Publications 

25.  R&D  Completion  Prior  to  IOT&E 
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1.  TEST  PLANS  AND  CRITERIA 


By  the  end  of  the  validation  phase,  test  and  evaluation  plans  and 

test  criteria  should  be  established  so  there  Is  no  question  as  to 

what  constitutes  a  successful  test  and  what  performance  Is  required. 

There  are  many  examples  where  the  objectives  of  a  given  test  were 
not  adequately  defined  or  the  criteria  for  success  delineated  clearly  in 
advance.  To  a  high  degree,  this  reduces  or  even  invalidates  the  value  of 
the  test  process.  Repeat  testing  necessitated  by  such  actions  is  not 
uncommon . 

The  test  plan  must  clearly  define  the  primary  and  secondary  objectives 
of  each  planned  aircraft  test.  The  test  environment,  the  performance  points 
to  be  tested,  the  instrumentation  needs,  the  data  collection  plan,  and  the 
data  reduction  requirements  must  be  generally  stated  sufficiently  in  advance 
of  any  test  being  made. 

In  summary,  the  test  program  for  the  aircraft  should  have  the  answers 
to  the  basic  7  "W's?" 

What  are  you  testing? 

Why  are  you  testing  it? 

When  are  you  testing  it? 

Where  are  you  testing  it? 

What  do  you  expect  to  find  out? 

What  will  you  do  if  test  results  meet,  exceed,  or  fall  short  of 
goals? 

What  is  the  effect  of  test  results  on  the  program  costs, 
schedules  and  the  system  capability? 

Review  of  the  above  can  assure  that  testing  remains  coupled  to  pro¬ 
gress  of  the  program  and  can  prevent  useless,  inconsequential  testing.  A 
formal  document  which  requires  the  answers  to  the  above  questions  would 
change  almost  any  test  program  ever  prepared — and  for  the  better. 

2.  MILESTONES  AND  GOALS 

Assure  an  integrated  system  test  plan  that  pre-establishes  milestones 
and  goals  for  easy  measurement  of  program  progress  at  a  later  time. 
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In  an  R&D  program  some  failures  may  be  expected,  but  the  plan  should 
be  structured  to  generally  ensure  that: 

(a)  Repetitive  failure  will  be  held  at  an  absolute  minimum  and 
each  substantive  failure  will  be  analyzed  and  the  cause 
corrected  before  subsequent  flight  tests. 

(b)  The  schedule  will  accommodate  problems. 

(c)  There  is  a  clear  statement  of  objectives  for  each  test. 

(d)  The  number  of  tests  that  are  expected  to  yield  an  answer 
have  been  identified. 

(e)  Qualification  testing  of  components  and  subsystems  should 
occur  at  the  earliest  possible  time  in  the  program.  Where 
existing  technology  is  involved,  qualification  can  usually 
occur  prior  to  the  onset  of  flight  testing.  Where  advanced 
technology  is  involved,  qualification  should  be  introduced 
just  as  soon  as  practicable  in  the  development  flight  test 
program — but  in  all  cases  before  production  of  operational 
hardware  is  undertaken,  provided  that  the  prototype  hard¬ 
ware  represents  the  production  hardware.  Component  or  sub¬ 
system  problems  that  are  detected  at  the  full  system  flight 
test  level  are  extremely  costly. 

(f)  Clear,  well-defined  milestones  for  review  and  commitment 
to  the  next  test  phase  have  been  defined. 

Almost  without  exception,  every  aircraft  program  has  experienced 
flight  test  failures  or  anomalies.  The  degree  to  which  corrective  action 
has  been  taken  prior  to  additional  flight  tests  has  varied  extremely  among 
various  programs.  The  motivation  for  continuing  flight  test  without  ade¬ 
quate  failure  diagnosis  and  corrective  action  is  associated  with: 

•  The  desire  to  claim  that  the  program  has  completed  its 
major  milestones. 

•  Failure  to  clearly  identify  test  objectives. 

S  •  A  superficial  conclusion  that  the  failure  was  random. 

•  Flying  of  unqualified  components  which  could  have  been 
qualified  before  flight. 

Although  milestones  are  normally  shown  on  a  time  schedule,  it  must 
be  understood  that  the  date  is  a  target  and  the  milestone  objectives  must 
be  accomplished  before  the  milestone  will  be  passed.  Cost  milestones  have 
little  meaning  unless  development  and  testing  progress  can  be  measured 
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accurately,  and  this  is  extremely  difficult  because  R&D  is  full  of  surprises 
and  unfolding  problems.  In  past  aircraft  programs  that  have  been  tied  to 
fiscal  or  date  milestones,  there  has  been  no  practical  way  of  readjustment 
of  the  program  short  of  keep  going,  stop  or  renegotiate.  Since  the  latter 
two  alternatives  are  drastic  measures,  usually  this  situation  led  to  placing 
production  orders  for  aircraft  in  spite  of  development  and  test  technical 
difficulties  at  the  time. 


3.  OPERATING  CONCEPTS  AND  ENVIRONMENT 

The  operational  concepts  and  the  environments  in  which  the  aircraft 
will  be  expected  to  operate  and  to  be  tested  in  OT&E  should  be 
specified. 


The  expected  operational  concepts  and  operating  environment  of  the 
aircraft  should  be  stated.  Aircraft  specifications  should  not  be  confined 
to  technical  items  such  as  speed,  weight,  turn  rate,  G's,  military  specifi¬ 
cation  compliance,  etc.,  but  also  should  provide  environmental  and  opera¬ 
tional  information  on  such  things  as  concepts  for: 

Operation:  If  the  aircraft  is  to  operate  in  the  tropics  or  desert, 
this  should  be  clearly  stated  and  aircraft  tests  planned  in  that 
environment.  If  the  aircraft  is  to  operate  off  of  sod  runways, 
this  should  be  stated  and  aircraft  tests  planned  under  these  con¬ 
ditions. 

Training  and  Personnel:  Simple  succinct  statements  as  to  the 
number  and  training  of  the  personnel  that  are  to  be  used  in  opera¬ 
tions  and  test  cycles  may  revise  the  design  and  sub-test  philo¬ 
sophies  and  eliminate  the  wide  disparities  of  results  on  aircraft 
programs  where  maintenance  or  other  environmental  considerations 
are  major  factors.  Few,  if  any,  aircraft  systems  that  have  been 
deployed  have  functioned  to  anywhere  near  the  level  of  pro¬ 
ficiency  that  is  experienced  when  unlimited  contractor  maintenance 
is  used. 

Logistics :  The  maintenance  and  supply  concept  to  be  used  should 
be  specified.  Such  a  thing  as  the  type  of  maintenance  support 
facilities  planned  can  be  an  important  design  consideration. 

Such  environmental  and  operational  information  would  have  assisted 
in  test  planning  in  past  programs  and  could  have  precluded  major  changes 
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in  some  present  aircraft  systems  and  operational  limitations  in  others. 

Without  a  positive  guide  to  the  likely  concepts  and  environment— relia- 
♦ 

bility,  maintainability,  availability  and  accuracy  may  not  be  inter¬ 
related  in  the  design  and  test  process. 

Stress  the  conduct  of  the  whole  system  test  and  particularly  the 
avionics  suite  test  in  high  temperature,  high  humidity  operating  condi¬ 
tions.  Although  new  aircraft  are  subjected  to  all-weather  testing,  the 
adequacy  of  the  test  is  dependent  upon  the  actual  weather  experienced 
during  the  scheduled  test  period.  In  the  past,  subsequent  operational  use 
has  exposed  unsatisfactory  conditions,  particularly  in  hot,  humid  climates. 
For  example,  in  one  major  aircraft  program  the  hot  weather  tests  were  run 
in  Panama  and  no  great  problems  were  encountered.  When  the  airplane  was 
deployed  to  SEA  considerable  avionics  trouble  was  caused  by  condensation. 

In  operations,  the  aircraft  stayed  at  high  altitude  long  enough  to  get  the 
airframe  cold  soaked.  After  landing,  the  high  humidity  resulted  in  con¬ 
siderable  condensation.  In  the  inadequate  tropic  development  testing, 
apparently  the  flight  profiles  had  insufficient  high  altitude  time  to 
set  up  this  condition. 

4.  PRIORITY  OF  REQUIREMENTS 

The  prime  driving  requirements  for  the  aircraft  mission  should  be 
stated  and  where  possible  priority  and  costs  should  be  associated 
with  each  requirement. 

Normal  development  requires  many  tradeoffs;  therefore,  priorities  for 
various  characteristics  are  necessary  for  guidance  in  this  process.  The 
contractual  documents  should  state  the  relative  importance  of  various  re¬ 
quirements,  e.g.,  ability  to  operate  in  stated  environments,  cost,  speed, 
maintainability,  etc.  However,  all  the  characteristics  must  be  carefully 
considered  in  any  compromises  even  though  they  may  not  be  top  priority. 

For  example,  a  weight  savings  program  on  one  aircraft  resulted  in  reducing 
the  structural  life  of  the  airframe  by  about  60  percent. 

The  priorities  can  serve  as  a  basis  for  scheduling  analyses,  simula¬ 
tions  and  flight  tests  of  the  aircraft  or  its  major  subsystems  and 
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components.  Because  T&E  is  expensive,  it  is  important  that  the  elements 
critical  to  the  decision  to  continue  the  program  be  tested  as  early  as 
practical.  For  example,  in  developing  a  T&E  plan  for  a  new  close  air 
support  aircraft  designed  to  offer  greater  bombing  accuracy  than  its  orede- 
cessors,  it  is  far  more  important  to  determine  the  actual  bombing  accuracy 
than  to  precisely  determine  expected  combat  radius. 

5.  TEST  PROGRAM  BUILDING  BLOCKS 

In  the  validation  phase,  demonstrate  that  the  high  risk  technology  is 
in  hand  and  in  planning  the  full-scale  development  test  program  ensure 
components  and  the  subsystems  are  adequately  qualified  for  incorpora¬ 
tion  into  system  tests. 

In  the  validation  phase,  necessary  testing  should  be  done  to  demon¬ 
strate  that  the  technology  is  in  hand  to  justify  full-scale  development. 

For  example,  in  a  bomber  program  competitive  system  studies  were  conducted 
to  adequately  define  the  conf iguration.  This  work  was  done  in  coordina¬ 
tion  with  competitive  engine  advanced  developments.  A  sizing  and  cycle 
matching  process  was  pursued  to  firm  up  on  probable  engine  definition.  At 
the  same  time,  core  engine  advanced  technology  was  used  as  a  basis  to 
proceed  with  design  of  hardware  for  test  engines  with  the  desired  by-pass 
ratios,  air  flow,  specific  fuel  consumption,  thrust,  thrust  augmentation 
system,  etc.  In  addition,  challenging  avionics  areas  were  studied  and 
hardware  was  built  to  investigate  high  risk  problems. 

In  planning  for  the  full-scale  development  test  program,  it  is  impor¬ 
tant  that  each  component  and  then  each  subsystem,  and  finally  the  full 
system  be  tested  in-turr  as  thoroughly  as  practical  on  the  ground.  Avoid 
including  untested  items  in  the  system  flight  tests.  In  aircraft  testing, 
normally  there  are  a  number  of  aircraft  instrumented  and  specialized  for 
testing  in  a  given  area.  For  example,  the  first  airplane  to  fly  is  usually 
the  stability  and  -’ontrcl-f light  qualities  aircraft.  Other  test  aircraft 
may  be  dedicated  to  such  things  as  ground  static  tests,  engine/airframe 
(propulsion)  tests,  avionic  tests,  ordnance  tests,  post  stall  tests,  and 
finally  full-system  tests.  Some  programs  will  have  more  than  one  test 
aircraft  devoted  to  an  area.  The  number  of  test  aircraft  in  the  program 
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vary  widely.  In  fighter  and  attack  aircraft  developments,  the  number 
of  aircraft  have  recently  varied  from  about  six  to  eighteen  test  aircraft 
in  the  basic  R&D  program.  A  large  cargo  airplane  program  had  five  R&D 
aircraft  and  two  heavy  bomber  developments  have  been  structured  around 
three  test  aircraft.  Whatever  the  number  of  test  aircraft,  the  flight 
test  program  should  concentrate  on  developing  a  safe  flying  machine  first 
and  then  selectively  adding  and  proving  the  major  subsystems.  Finally, 
the  whole  system  should  be  scheduled  for  the  testing,  debugging,  retest¬ 
ing,  and  finally  the  demonstration  phase. 

6 .  TECHNOLOGY  CONCEPTS 

Each  concept  to  be  used  in  the  aircraft  system  (e.g.,  aerodynamics, 
structures,  propulsion)  should  be  identified  and  coded  according 
to  prior  application,  prior  or  future  research;  tests  for  each  of 
the  concepts  should  be  specified  with  the  effect  of  failure  identi¬ 
fied. 

A  formal  listing  of  all  concepts  planned  for  use  in  an  aircraft 
system  will  provide  understanding  for  all  parties  and  enhance  the  use  of 
experience  from  prior  tests.  Also  the  setting  out  of  the  basis  for  appli- 
catior  of  the  concept  can  aid  in  eliminating  wishful  thinking  related  to 
new  approaches  and  at  the  same  time  provide  an  index  to  the  testing  com¬ 
munity  of  what  needs  to  be  tested  in  lieu  of  only  documents  of  what  is 
planned  for  test.  For  example,  the  application  of  a  new  rotor  system 
concept  led  to  serious  technical  problems  in  one  program  and  eventual 
cancellation  of  the  development.  Since  no  prior  similar  (in  scale)  appli¬ 
cation  existed  for  the  rotor  concept,  the  concept  should  have  been  coded 
for : 

•  Tests  planned — subscale  and  full  scale. 

•  Effect  of  failure — weight,  cost,  time,  operational  capability. 

•  Contractual  arrangements  to  allow  early  and  expeditious  change 
of  concept  in  the  event  of  difficulty. 

•  Whether  the  customer  set  aside  sufficient  funds  to  match  a 
reasonable  percentage  of  failures  or  concept  changes. 
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DT&E/IOT&E  PLAN 


The  aircraft  DT&E/IOT&E  test  plan  should  be  reviewed  to  ensure  it 
includes  ground  and  flight  tests  necessary  to  safely  and  efficiently 
develop  the  system. 


Desirable  features  to  look  for  in  the  aircraft  DT&E/IOT&E  test  plan 
include: 

(a)  Piece  part,  component,  and  subsystem  development  qualifications 
in  that  order  before  including  them  in  the  aircraft  for  system 
testing. 

(b)  Before  the  first  flight  structural  tests  of  the  major  airframe 
assemblies  are  made,  engine/inlet  nozzle  compatibility  tests 
are  conducted  in  wind  tunnels,  and  the  engine  must  be  at  least 
successful  through  the  Preliminary  Flight  Rated  Tests  (PFRT) . 

(c)  Continued  airframe  static  testing  of  critical  conditions  are 
desirable  to  lead  broader  flight  envelope  exploration.  Ground 
fatigue  tests  of  the  airframe  should  start  as  soon  as  the  pro¬ 
duction  airframe  configuration  is  determined.  There  is  con¬ 
siderable  value  in  continuing  these  until  destruction. 

(d)  Bench  tests  and  system  integration  lab  tests  of  the  avionics 
and  possibly  mobile  ground  tests  of  the  navigation  gear  and 
air  tests  of  the  prototype  avionics  gear  are  desirable  before 
initial  airborne  avionics  tests  in  the  new  aircraft. 

(e)  Ordnance  ground  testing  should  precede  the  ordnance  flight 
testing. 

(f)  Military  Preliminary  Evaluations  (MPEs)  should  be  conducted 
as  soon  as  practical  so  that  user/evaluator  inputs  are  avail¬ 
able  early  and  necessary  corrections  of  deficiencies  can  be 
made  during  the  R&D  phase. 

(g)  All  qualified  systerj,  including  airframe,  engine,  crew  escape 
system,  avionics  and  the  ordnance  system,  are  combined  for 
final  system  tests  and  demonstrations. 

(h)  2000-3000  hours  of  flight  test  on  a  sophisticated  military 
airplane  are  usually  necessary  for  the  contractor  to  be  ade¬ 
quately  prepared  to  run  a  few-hundred-hour  demonstration 

to  satisfy  the  government  that  the  basic  R&D  is  complete. 

(i)  The  test  plan  must  also  include  scheduling  of  manning,  train¬ 
ing,  and  conduct  of  the  IOT&E  together  with  scheduling  the 
required  technical  manuals,  ground  support  equipment,  test 
aircraft  and  test  location  facilities. 
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8.  TEST  FAILURES 


Make  T&E  plans  assuming  failures — they  are  Inevitable. 

DT&E  and  OT&E  test  plans  should  Include  time  and  resources  necessary 
for  investigating  test  failures  and  making  provisions  for  elimination  of 
the  cause  before  the  next  test  of  that  failed  system.  Although  there  are 
some  exceptions,  the  general  rule  that  no  additional  ma.1or  tests  will  pro¬ 
ceed  until  a  vital  failure  is  understood  and  fixed,  should  be  observed. 

In  sjiije  aircraft  programs,  failures  have  occurred  which  were  not 
adequately  diagnosed  before  further  flight  tests  were  made  and  the 
failures  repeated.  For  example,  if  an  air-to-air  missile  damages  the  air¬ 
plane  during  launching,  flight  tests  not  involving  missile  launch  could 
be  continued,  but  if  the  flight  control  hydraulic  systems  failed,  the  air¬ 
craft  would  be  subject  to  grounding  for  safety  of  flight  reasons  until 
fixes  were  made.  In  all  instances  of  substantial  failure,  it  is  better 
to  spend  time  in  careful  diagnosis  and  if  necessary,  in  changing  subse¬ 
quent  flight  tests  and  their  schedules  than  to  assume  the  same  failure 
will  not  recur. 

There  should  be  back-up  tests  planned  so  that  failure  of  one  item 
will  not  result  in  unproductive  delay.  The  cost  of  test  personnel  and 
facilities,  dedicated  to  a  program,  continues  even  though  a  specific 
test  program,  because  of  a  failure,  may  be  placed  in  a  stand-by  status. 

In  summary,  experience  has  shown  that  there  has  been  a  marked  tend¬ 
ency  to  be  over  optimistic  in  planning  time  and  resources  for  test. 

9.  JOINT  TESTING 

When  a  new  aircraft  development  program  requires  joint  testing 
during  OT&E,  prior  to  DSARC  II,  the  test  plan  should  include  the 
type  c  i  tests  and  resources  required  from  other  activities  and 
services , 

Several  weapon  systems  designed  for  defense  penetration  or  sup¬ 
pression  have  had  inadequate  testing  and  evaluation  in  relation  to  their 
mission  objective. 
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In  general,  the  T&E  plan  should,  as  far  as  possible,  include  offense/ 
defense  engagements  representative  of  those  in  which  the  new  system  is 
expected  to  operate.  Many  programs  have  not  given  full  attention  to  this. 
Offense/defease  testing  may  be  addressed  in  several  phases,  such  as: 

(a)  Testing  against  the  best  simulation  of  the  assumed  threat 
which  can  be  made  available,  either  in  the  field  or  in  a 
laboratory  with  variations  in  the  threats,  scenarios,  and 
environment  to  cover  the  range  expected  in  combat. 

(b)  Testing  against  advanced  U.S.  technology  which  may  be  repre¬ 
sentative  of  a  potential  threat. 

(c)  Testing  against  electronic  countermeasures  must  be  investi¬ 
gated.  In  the  Conceptual  Phase  the  matter  of  countermeasures 
must  be  considered  to  ensure  that  the  development  is  sound. 

During  the  development,  countermeasures  are  a  secondary  con¬ 
sideration;  however,  after  the  new  system  is  developed,  it  is 
necessary  to  ensure  that  it  can  be  used  in  the  presence  of 
countermeasures  by  the  use  of  technical  features  of  the  hard¬ 
ware/software  or  by  alternate  operating  modes  or  different 
tactics.  Therefore,  lacer  phases  of  OT&E  testing  should 
include  appropriate  electronic  countermeasure  aspects. 

In  summary,  there  is  considerable  evidence  that  weapon  systems  have 
been  less  satisfactory  under  combat  conditions  than  earlier  testing  pro¬ 
mised.  Had  test  plans  been  more  realistic  with  better  field  simulations, 
the  results  would  have  been  more  meaningful.  This  in  turn  may  demand 
joint  Service  involvements.  For  example,  prototype  tanks,  armored  per¬ 
sonnel  carriers  and  advanced  anti-aircraft  equipments  and  operators  will 
be  required  for  joint  operational  testing  of  a  new  close  air  support 
aircraft . 


.10.  TRACEABILITY 

The  aircraft  development  and  test  program  should  be  designed  and 
scheduled  in  such  a  way  that  if  trouble  arises  the  source  of  the 
trouble  can  be  traced  back  through  the  lab  tests  and  the  analytical 
studies. 


The  testing  and  instrumentation  plans  should  be  designed  to  allow 
correlation  and  correction  of  lab  tests  and  analytical  studies.  Aircraft 
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programs  have  been  designed  for  success,  therefore  when  a  major  variance 
was  encountered,  the  level  of  ambiguities  created  by  the  test  set-up  and 
the  instrumentation  sometimes  precluded  a  simple  impact  analysis.  In  many 
cases  the  analysis  could  not  be  corrected  and  improvement  could  not  be  made 
without  additional  testing  or  instrumentation.  Thus,  the  scheduling  of 
analyses,  lab  tests,  hardware  fabrication  and  full-scale  tests  should  be 
such  that  the  data  from  each  activity  can  be  constructively  used  in  the 
next  step  and  traced  back  thiough  previous  steps.  If  measurements  are  be¬ 
ing  made  on  static  or  fatigue  tests,  strain  gauge  the  flight  article  in  the 
same  way  and  in  the  same  locations,  particularly  on  indeterminate  structures. 

Also,  the  wind  tunnel  test  points  should  include  the  ones  that  are  needed 
to  correlate  with  analyses  and  lab  tests.  As  the  aircraft  system  goes 
from  the  lab  to  full-scale,  the  instrumentation  should  provide  for  maxi¬ 
mum  practical  traceability  all  the  way  back  to  the  analytical  studies. 

11.  COMPETITIVE  PROTOTYPE  TESTS 

When  a  competitive  prototype  test  program  is  used,  the  aircraft  should 
be  compared  on  the  basis  of  the  performance  of  critical  missions 
using  both  test  and  operational  crews. 

The  aircraft  should  be  compared  on  carefully  prepared  criteria.  It 
might  include  basing  and  runway  requirements,  maintenance  efforts  per 
flight  hours,  flight  performance  characteristics,  survivability,  ordnance 
carrying  capability,  weapon  delivery  accuracy,  and  cost.  To  the  extent 
possible,  aircrews  for  fly-offs  should  be  a  mix  of  operational  and  test 
pilots,  and  it  would  be  desirable  for  some  of  the  operational  crews  to  have 
had  experience  with  the  aircraft  that  the  prototype  is  designed  to  replace. 

In  an  attack  aircraft  prototype  fly-off,  tactical  crews  were  used  and  their 
evaluations  provided  valuable  inputs  for  the  overall  aircraft  comparison. 

12.  PROTOTYPE  SIMILARITY  TO  DEVELOPMENT  AND  PRODUCTION  AIRCRAFT 

A  firm  determination  should  be  made  of  the  degree  of  similarity  of 
the  winning  prototype  (in  a  competitive  prototype  program)  to  the 
development  and  production  aircraft  in  order  that  test  results 
derived  from  the  prototype  in  the  interim  period  prior  to  avail¬ 
ability  of  the  engineering  development  aircraft  can  be  utilized 
most  effectively. 
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In  competitive  prototype  programs,  the  winning  prototype  will  be 
used  for  tests  for  a  period  prior  to  the  availability  of  engineering 
development  aircraft.  In  order  to  assure  the  validity  of  tests  in  areas 
such  as  drag  reduction,  stall/post  stall/spin  and  weapons  compatibility 
on  the  prototype  and  the  applicability  of  these  results  for  the  develop¬ 
ment  and  production  aircraft,  the  differences  of  the  aircraft  must  be 
understood.  For  example,  in  an  attack  aircraft  program  the  engineering 
development  model  incorporated  a  repositioned  landing  gear  and  aerodynamic 
slats  in  place  of  fixed  slats. 

13.  PROTOTYPE  TESTS 

The  prototype  aircraft  test  data  should  be  used  to  determine  where 
emphasis  should  be  placed  in  the  engineering  development  program. 

Carefully  review  prototype  test  data  to  ensure  full  use  is  made  of 
the  data  to  plan  the  subsequent  development  and  tests. 

The  prototype  tests  will  undoubtedly  show  deficiencies  which  should 
be  corrected.  For  example,  one  successful  prototype  showed  that  the 
engine  inlet/wing  arrangement  resulted  in  an  intake  airflow  reduction  in 
a  pull-out  that  tended  to  stall  the  engine.  In  the  early  prototype  tests 
this  undesirable  characteristic  was  handled  by  a  fuel  control  expedient. 
The  fix  later  in  the  prototype  test  program  consisted  of  a  wing  slat  to 
restore  proper  intake  air  flow  at  the  critical  condition  so  that  a  power 
loss  was  not  necessary.  The  prototype  tests  will  be  helpful  in  planning 
and  conducting  the  full  scale  development. 

If  the  level  of  flight  safety  risk  is  low,  a  sole  prototype  may  be 
reasonable  to  demonstrate  a  system  where,  for  example,  the  avionics  tie- 
in  might  be  the  prime  reason  for  prototyping.  On  the  other  hand,  the  risk 
may  be  high  to  prototype  only  one  new  concept  V/STOL  aircraft. 

14.  INLET/ENGINE/NOZZLE  MATCH 

The  aircraft  test  program  should  provide  for  early  and  adequate 
inlet /engine/nozzle  match  through  a  well  planned  test  program  with 
time  programmed  for  corrections. 
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The  inlet/engine/nozzle  match  is  the  major  match  in  aircraft  and  is 
often  critical  in  obtaining  the  required  aircraft  performance.  Most  high 
performance  jet  fighter  aircraft  have  experienced  unsatisfactory  inlet/ 
engine/nozzle  conditions  during  development;  this  resulted  in  engine  stalls 
under  some  rapid  throttle  operation  situations  and  high  drag  in  some  con¬ 
ditions.  Full-scale  inlet/engine/nozzle  wind  tunnel  tests  over  the  expected 
operational  range  of  angle  of  attack,  speed,  and  altitude  have  been  use¬ 
ful  in  identifying  problems  and  developing  fixes  so  as  to  achieve  low  drag, 
stable,  stall-free  engine  operation,  both  in  steady  state  and  transient 
conditions  where  rapid  throttle  bursts  and  chops  and  after-burner  light 
offs  and  shutdowns  were  attempted  throughout  the  performance  envelope. 

Late  changes  of  large  magnitude  in  bleed  air  requirements  for  boundary 
layer  control,  air  conditioning,  and  cabin  pressurization  can  invalidate 
earlier  inlet /engine /nozzle  tests  and  should  either  be  defined  early  or 
considered  in  the  range  of  test  conditions. 

15.  SUBSYSTEM  TESTS 

There  should  be  a  balanced  program  for  the  aircraft  subsystem  tests. 

The  test  program  for  subsystems  may  be  structured  using  working  mock- 
ups  for  test  of  such  items  as  flight  control  and  landing  gear  and  flying 
test  beds  for  the  fire  control,  engines  and  any  special  area  of  technology 
such  as  a  super-critical  wing.  However,  the  cost  of  the  subsystem  test 
program  has  to  be  balanced  against  the  possible  cost  you  might  incur  if 
you  did  not  do  the  testing  and  subsequently  subsystems  deficiencies  show 
up  in  the  full  system  tests. 

For  a  subsystem  such  as  an  avionics  package  that  may  be  available  in 
prototype  form  well  before  a  prototype  aircraft  is  available,  the  subsystem 
may  be  flown  in  alternate  aircraft  (c.g.,  an  avionics  system  mounted  in  an 
aircraft  that  will  simulate  the  operating  environment  as  closely  as  possi¬ 
ble).  Bench  testing  of  avionics  components  and  system  integration  labora¬ 
tory  testing  of  avionics  are  necessary,  but  results  must  be  supported  by 
data  obtained  under  operational  conditions  in  the  flight  environment. 
Subtleties  often  show  up  only  in  flight  testing.  The  airframe  should  be 
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flown  at  the  earliest  practical  date  and  should  not  be  held  up  awaiting 
avionics , 

16.  PROPULSION  SYSTEM 

If  the  aircraft  program  Is  paced  by  the  propulsion  system  development, 
an  early  advanced  development  project  for  the  propulsion  may  be 
appropriate  for  a  new  concept. 

The  progress  in  the  development  of  the  propulsion  system  is  normally 
the  key  factor  in  determining  when  full-scale  aircraft  development  is 
warranted.  However,  when  other  subsystems  such  as  the  avionics,  are  also 
critical  and  when  there  is  a  time  urgency  to  get  the  new  system,  there 
have  been  isolated  examples  where  the  aircraft  system  development  program 
has  been  initiated  with  an  interim  power  plant.  This  requires  additional 
engineering  and  testing  and  can  only  be  justified  in  unusual  cases.  Tests 
which  use  the  substitute  power  plant  should  be  restricted  to  providing  a 
platform  for  other  subsystem  flight  tests.  Full  system  tests  should  be 
planned  only  with  the  proposed  power  plant. 

If  the  new  aircraft  is  designed  to  use  a  new  power  plant,  it  is 
desirable,  during  the  Validation  Phase,  at  least  to  demonstrate  success¬ 
ful  operation  of  the  core  engine.  The  specific  matching  of  the  complete 
engine  and  the  airframe  and  the  resulting  selection  of  by-pass  ratio, 
augmentation,  configuration,  etc.  should  be  expedited  so  that  the  engine 
can  be  configured  and  the  propulsion  engineering  development  initiated 
early. 

17 .  EMI  TESTING 

Full-scale  aircraft  systems  tests  in  an  anechoic  chamber  are  desir¬ 
able  for  some  aircraft. 

E-type  aircraft  with  sophisticated  avionics  and  Electronic  Warfare 
Systems  (EWS)  are  prime  candidates  for  EMI  full-scale  testing  in  an 
anechoic  chamber.  One  program  tested  over  20,000  operating  combinations 
in  this  manner  and  fixed  120  interferences.  This  kind  of  analysis  would 
have  been  much  more  costly  and  time  consuming  in  flight  test. 
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EMI  test  plans  for  all  aircraft,  but  particularly  for  EWS  airplanes, 
should  be  reviewed  to  ensure  that  the  anecholc  chamber  testing  Is  being 
used  where  practical. 

18.  PARTS  INTERCHANGE 

Early  plans  should  provide  for  tests  where  theoretically  Identical 
parts,  particularly  in  the  avionics,  are  Interchanged  to  ensure 
that  aircraft  systems  can  be  maintained  In  readiness. 

Poorly  designed  gear  or  parts  not  fully  developed  can  sometimes 
be  made  to  operate  properly  by  fine  adjustment. 

Test  plans  must  ensure  that  poor  design  is  not  hidden  by  "fine 
tuning"  or  "tweaking"  of  parts  and  components  of  the  aircraft  and  the 
avionics.  If  this  is  permitted,  it  is  unavoidable  that  future  redesign 
will  be  required.  Furthermore,  these  tests  should  provide  for  the  eval¬ 
uation  of  time  required  to  put  the  aircraft  or  subsystem  in  full-up 
readiness  status  using  the  planned  size  and  skills  of  the  field  crew. 

These  tests  are  particularly  important  in  aircraft  to  (a)  assess  main¬ 
tainability,  (b)  to  determine  if  there  are  adequate  interface  space 
tolerances,  and  (c)  to  ensure  EMI  tolerance  to  maintenance  and  parts 
replacement . 

19.  HUMAN  FACTORS  DEMONSTRATIONS 

Ensure  adequate  demonstration  of  human  factors  is  considered  in  the 
test  plan. 

At  an  appropriate  time  in  the  concept  definition  or  the  development 
phase,  T&E  should  ensure  that  the  human  factor  concepts  embodied  in  the 
proposed  system  design  are  tested.  Questions  of  safety,  comfort,  appro¬ 
priateness  of  man-machine  interfaces,  reaction  time,  performance  under 
stress,  fatigue,  and  number  and  skill  levels  of  the  personnel  required  and 
many  other  factors  must  be  examined.  Testing  early  versions  in  the 
"human  acceptability  and  compatibility"  environment  is  extremely  important. 
This  will  not  only  result  in  better  performance,  but  will  also  help  to 
reduce  (and  validate)  requirements. 
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For  example,  retargeting  or  selection  of  alternate  weapons  under 
battle  conditions  may  be  limited  by  human  factors.  Early  test  can  bring 
out  deficiencies  which  redesign  can  then  correct.  The  maintainability 
of  any  system  also  is  clearly  of  great  importance.  Testing  to  prove 
maintainability  form  the  human  standpoint  is  useful. 


20.  CONTRACT  FORM 

The  contract  form  can  be  extremely  important  to  the  T&E  aspects. 


In  fixed  price  aircraft  development  contracts  there  were  pressures 
to  force  the  contractors  to  obtain  specified  performance  and  characteris¬ 
tics  regardless  of  what  was  learned  as  the  developments  progressed. 

Hindsight  now  indicates  that  this  was  not  sensible  in  a  number  of  cases. 

In  the  Validation  Phase  the  contracting  strategy  is  usually  established 
in  the  RFP  and  the  subsequent  evaluation  and  source  selection  process. 

From  the  T&E  standpoint,  the  contract  should: 

(a)  Permit  early  user  and  evaluator  participation. 

(b)  Establish  incentives  only  after  careful  consideration  and 
resolution  of  possible  undesirable  aspects  from  the  T&E 
standpoint.  The  incentives  should  not  be  based  on  extreme 
corners  of  the  theoretical  performance  envelope  unless 
there  is  a  high  operational  payoff.  On  the  othei-  hand, 
the  incentive  should  not  constrain  the  developer  from 
exploring  the  likely  operational  performance  envelope. 

(c)  Facilitate  engineering  changes  resulting  from  knowledge 
gained  during  the  test  program  and  from  threat  changes. 


21.  GOVERNMENT  FURNISHED  EQUIPMENT 

If  there  are  GFE  and  other  government  commitments  in  the  proposed 
contract,  there  should  be  a  clear  statement  of  responsibility  and 
liability  of  the  interface  of  the  GFE  and  the  development  aircraft 
because  of  the  impact  on  test  schedules  and  performance. 


Relative  to  the  impact  on  testing,  special  attention  should  be 
given  to  the  following: 
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(a)  Can  the  gear  with  required  performance  be  available  for  test 
when  required?  The  most  prominent  aircraft  GFE  items  are 
engines  and  ECM, 

(b)  Can  government-supported  facilities  provide  the  T&E  assis¬ 
tance  required  at  the  time  needed?  If  not,  is  it  reasonable 
to  construct  the  required  facilities  (test  range,  instru¬ 
mentation,  building,  etc.)?  If  not,  what  alternatives 

are  available? 

(c)  Avoid  contract  terms  on  fixed  price  contracts  that  vaguely 
commit  the  government.  Do  not  include  "government  support 
as  required"  or  "test  facilities  will  be  made  available  when 
needed. " 

The  responsibility  for  the  interface  of  GFE  and  development  aircraft 
must  be  precisely  structured.  This  interface  agreement  should  not  be 
merely  in  fixed  terms.  It  should  include  a  contractual  mechanism  to  allow 
the  movement  of  funds  from  contract  to  contract  without  time  lag  to  allow 
the  most  cost  effective  solution  to  the  interface  problems. 


22.  MILITARY  PRELIMINARY  EVALUATION 

Adequate  resources  should  be  scheduled  for  the  aircraft  Military 
Preliminary  Evaluation  (MPE)  and  a  positive  program  should  exist 
for  the  utilization  of  MPE  information  at  the  time  of  IOT&E. 


The  Military  Preliminary  Evaluations  provide  an  opportunity  for 
service  crews  to  make  an  early  evaluation  of  an  airplane.  Flights  by 
military  evaluation  personnel  early  in  DT&E  may  uncover  deficiencies  re¬ 
lated  to  operational  use  of  the  system.  These  tests  must  be  early  so  that 
there  is  time  remaining  in  the  development  program  to  correct  the  problems 
discovered.  Emphasis  during  MPEs  should  be  on  fundamentals  (Is  it  safe? 
Does  it  show  promise  of  providing  improved  capability?  What  are  the  gross 
deficiencies?)  rather  than  on  performance  guarantee  items  (maximum  speed, 
weapons  delivery  accuracy,  etc.).  For  example,  in  one  aircraft  program, 
the  MPE  found  that  the  aiming  display  on  the  windscreen  was  too  narrow 
for  operational  use.  During  subsequent  development,  this  deficiency  was 
easily  corrected  and  retested.  If  this  deficiency  had  not  been  found  until 
IOT&E,  at  the  end  of  the  basic  R&D  program,  the  correction  could  have 
delayed  the  initial  production  program  or  could  have  been  a  costly  mod 
program.  Consideration  planning  should  be  done  to  get  the  most  out  of  the 


23.  USER  PARTICIPATION 


It  is  imperative  that  the  operational  command  actively  participate 
in  the  DT&E  phase  to  ensure  that  the  user  needs  are  represented  in 
the  development  of  the  system. 

Where  user  participation  in  DT&E  has  been  inadequate,  additional 
problems  have  been  discovered  when' the  program  progressed  to  the  opera¬ 
tional  test  phase.  Some  deficiencies  discovered  late,  such  as  human 
factors  problems,  technical  manual  errors,  and  design  inadequacies,  could 
have  been  much  more  easily  corrected  if  they  had  been  found  early  in  the 
DT&E  by  user  part  icipants. 

Initially,  the  operational  test  command  should  play  an  advisor  role 
during  the  feasibility  and  engineering  testing,  and  finally  take  over 
leadership  in  the  conduct  of  the  operational  testing  program.  This  user 
participation  in  DT&E  should  facilitate  the  necessary  communication  and 
interaction  between  the  developing  and  operational  commands — especially 
needed  during  the  DT&E  and  IOT&E  phases. 

Some  aircraft  programs  in  the  past  have  not  provided  for  operators 
early  in  the  program  with  the  result  that  when  the  operators  flew  the  air¬ 
craft,  items  such  as  bad  displays,  and  poor  handling  characteristics  had 
not  been  identified.  In  one  program,  the  testing  was  conducted  in  such 
a  way  that  testers  and  users  were  not  permitted  to  communicate. 

24.  MAINTENANCE  AND  TRAINING  PUBLICATIONS 

The  aircraft  development  program  should  provide  for  concurrent 
training  of  crews  and  for  preparation  of  the  draft  technical 
manuals  to  be  used  by  IOT&E  maintenance  and  operating  crews. 

IOT&E  should  be  conducted  by  maintenance  and  operating  crews 
uitlizing  as  nearly  as  possible  the  type  of  materiel  and  information  that 
will  be  available  for  the  operational  unit.  Therefore  a  timely  program 
should  be  provided  to  train  maintenance  and  operating  crews.  Steps 
should  be  taken  to  ensure  that  the  required  materiel  (ground  support 
equipment,  technical  manuals,  parts  lists,  etc.)  will  be  available. 
Finally,  the  IOT&E  program  must  be  planned  so  that  typical  operations 
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can  be  tested  and  sufficient  data  gathered  to  analyze  the  results, 
main  conclusions  of  the  IOT&E  will  be  important  in  determining  the 
appropriateness  of  moving  the  programs  into  substantial  production  and 
deployment . 

25.  R&D  COMPLETION  PRIOR  TO  IOT&E 

The  testing  plans  should  ensure  that  before  an  aircraft  system  is 
subjected  to  IOT&E,  the  sub-systems  essential  to  the  basic  mission 
have  completed  basic  R&D. 

Development  problems  concerning  computers  and  their  software,  for 
example,  should  be  solved  during  DT&E.  Development  problems  carried  over 
into  OT&E  impede  operational  suitability  testing.  Further,  development 
problems  addressed  under  relatively  austere  field  conditions,  away  from 
laboratory  personnel  and  their  test  equipment,  are  more  difficult  to  solve. 

However,  in  parallel  with  the  IOT&E,  it  is  desirable  to  continue 
the  R&D  effort  with  the  instrumented  aircraft  on  alternate  missions  and 
degraded  back-up  mode  testing.  For  example,  in  an  attack  fighter  develop¬ 
ment,  the  problems  that  developed  in  the  operational  testing  effort  were 
quickly  reflown  on  the  instrumented  R&D  aircraft.  The  diagnosis  was 
easy  with  the  proper  data  from  the  instrumented  aircraft. 


III.  FULI  JALE  ENGINEERING  DEVELOPMENT  PHASE 

In  this  phase,  the  T&E  plans  developed  in  the  Validation  Phase  will 
be  refined  and  the  development  testing  conducted.  IOT&E  plans  will  similar¬ 
ly  be  refined;  personnel  will  be  assigned  and  trained  and  the  IOT&E  con¬ 
ducted. 

It  is  suggested  that  the  checklists  for  the  previous  phases,  parti¬ 
cularly  the  Validation  Phase,  be  reviewed  by  those  interested  in  a  check¬ 
list  for  full-scale  development.  The  full-scale  development  checklist 
includes : 

1.  Test  Design 

2.  Data  for  Alternate  Scenarios 

3.  Test  Milestones 

4.  Production  Engineering  Influence  on  R&D  Hardware 

5.  Running  Evaluation  of  Tests 

6.  Simulation 

7.  Software  Tests 

8.  Avionics  Mock-up 

9.  Escape  System  Testing 

10.  Structural  Testing 

11.  Gun  Firing  Tests 

12.  Post  Stall  Characteristics 

13.  Special  Attention  Items 

14.  Sub-System  Performance  History 

15.  Flight  Deficiency  Reporting 

16.  Crew  Limitations 

17.  Use  of  Operational  Personnel 

18.  Role  of  the  User 

19.  Ctew  Fatigue  and  System  Effectiveness 

20.  Test  Constraints  on  Crew 

21.  Complete  Basic  DT&E  Before  Starting  IOT&E 

22.  Realism  in  Testing 

23.  Test  All  Planned  Profiles  and  Modes 
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24.  Update  of  Operational  Test  Plans 

25.  Conduct  IOT&E  Early 

26.  Missile  Launch  Tests 

27.  Operational  Test  Realism 

28.  Mission  Completion  Success  Probability 
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TEST  DESIGN 


Test  programs  should  be  designed  to  have  a  high  probability  of 
Identifying  major  deficiencies  early,  during  DT&E  and  IOT&E. 

T&E  monitors  should  assure  that  testing  is  emphasized  in  those  areas 
where  major  deficiencies  are  most  likely  to  occur,  such  as  where  the 
state-of-the-art  has  been  significantly  advanced  in  propulsion  or  avionics. 

Test  programs  should  ideally  provide  time  for  fixes  to  be  developed 
and  engineering  change  proposal  (ECP)  action  to  be  initiated  in  the  R&D 
program  when  there  is  little  hardware  needing  retrofit. 

Later  in  system  acquisition,  when  a  substantial  quantity  of  air¬ 
craft  has  been  produced  and  delivered  to  the  user,  correction  of  major 
deficiencies  and  modifications  becomes  more  difficult.  These  usually 
involve  costly  retrofit,  in  addition  to  interim  operational  limitations 
imposed  on  the  user  which  can  impede  the  performanct  of  his  mission. 
Further,  these  retrofit  programs  are  funded  by  aircraft  modification 
money.  Modification  funds  are  always  in  short  supply  with  competition 
from  other  aircraft  systems,  safety  of  flight  items  receive  top  priority. 

There  have  been  two  examples  of  identifying  deficiencies,  one  good 
and  one  bad,  related  to  the  head-up  display  (HUD).  On  one  aircraft 
early  MPEs  recommended  a  broader  field  for  the  HUD.  This  change  was 
easily  made  in  the  DT&E  program.  In  another  program,  a  desirable  HUD 
change  to  display  more  information  was  not  recommended  until  after  deploy¬ 
ment.  At  that  point,  the  cost  to  change  was  high  and  the  improvement 
in  capability  unsuccessfully  competed  for  the  limited  aircraft  modifica¬ 
tion  funds  against  higher  priority  safety  of  flight  items  on  other  air¬ 
craft,  as  well  as  Increased  operational  capability  modification  on  other 
aircraft. 

2.  DATA  FOR  ALTERNATE  SCENARIOS 

Maximize  the  utility  of  the  test  data  gathered  by  careful  attention 
to  testing  techniques;  aircraft  instrumentation;  range  instrumenta¬ 
tion;  and  data  collection,  reduction,  and  storage. 
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There  are  an  infinite  number  of  scenarios  that  may  become  of 
interest  in  the  use  of  the  systems  and  the  operational  testing  can  only  be 
conducted  on  a  few  scenarios.  The  challenge  to  the  test  planners  and 
conductors  is  to  design  the  program,  the  instrumentation,  and  the  data 
collection  so  that  basic  information  gathered  on  the  selected  tests  can 
also  be  used  to  reasonably  estimate  what  the  system  would  actually  do  on 
alternative  missions.  For  example,  a  few  of  the  many  factors  which  might 
be  of  particular  interest  from  the  combined  DT&E  and  OT&E  testing  are: 

Readiness  -  Ability  of  airmen  to  maintain  aircraft 

on  alert  under  all  weather,  all  climate 
conditions  with  minimum  base  support 
facilities  as  well  as  with  normal  base 
facilities  and  logistical  support. 

Performance  -  Take-off  characteristics  from  appropriate 

surfaces  and  engine  out  characteristics 
at  various  altitudes  and  temperatures  with 
selected  loads  representing  the  allowable 
envelope. 

-  Best  miles/pound  of  fuel  at  different 
altitudes  at  various  stores  ranging  from 
clean  loads  to  high  drag  loads  and  without 
external  loads. 

Target  Acquisition  -  Ability  to  acquire  representative  classes 

of  targets  with  a  spread  of  acquisition 
assistance  in  varied  weather  and  day/ 
night  conditions. 

Survivability  -  Ability  to  destroy  or  to  survive  typical 

classes  of  fighter  and  ground  based 
defenses.  This  would  include  counter¬ 
measures  considerations. 

CEP  -  Delivery  accuracy,  with  representative 

items  from  various  classes  of  ordnance, 
from  typical  attack  conditions. 

Destruction  -  Measure  kill  potential  on  appropriate 

classes  of  targets  with  a  variety  of 
weapons  and  CEPs  if  it  has  not  been  done 
already. 

Recocking  -  Turn-around  time  with  various  loads, 

support,  and  weather. 
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The  OT&E  tests  should  minimize  repeats  of  DT&E  tests,  as  has  sometimes 
been  done  in  the  past,  and  should  strive  to  expand  the  tactical  and 
operational  aspects  to  include  other  conditions  and  points  within  the 
operating  capabilities  of  the  system.  These  data  would  be  used  to  provide 
a  broad  base  of  information  to  estimate  system  performance  in  alternate 
scenarios. 

3.  TEST  MILESTONES 

Development  programs  should  be  built  around  testing  milestones,  not 
calendar  dates. 

Testing  milestones  are  more  credible  measures  of  progress  of  a 
development  program  than  target  dates.  Where  testing  is  to  be  serially 
accomplished,  the  completion  of  a  critical  phase  is  more  meaningful  than 
meeting  an  artifically-established  calendar  date.  When  a  test  plan  is 
based  on  target  dates,  there  is  a  tendency  to  compress  the  testing  schedule 
should  test  slippage  occur.  This  tends  to  compromise  both  quality  and 
quantity  of  test  results. 

For  example,  a  target  date  was  specified  for  completion  of  the  150- 
hour  qualification  test  (QT)  for  the  power  plant  of  a  new  aircraft  weapon 
system.  The  date  seemed  to  have  been  artificially  selected  to  apply 
pressure  to  the  contractor  and  the  developer  in  the  interest  of  early 
qualification,  although  it  should  be  noted  that  the  same  engine  core  was 
to  be  used  in  another  weapon  system  which  had  slipped  its  development 
schedule.  The  target  date  did  apply  pressure  to  the  developer  and  the 
contractor,  but  when  they  were  unable  to  qualify  the  engine  by  the 
established  date,  there  was  no  apparent  impact — only  an  extension  to 
the  target  date  for  QT  completion. 

A.  PRODUCTION  ENGINEERING  INFLUENCE  ON  R&D  HARDWARE 

Encourage  that  production  philosophy  and  production  techniques 
be  brought  into  an  early  phase  of  the  design  process  for  R&D 
hardware  to  the  maximum  extent  practical. 
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There  have  been  aircraft  programs  in  which  major  components  have 
not  been  qualified  until  long  after  production  was  started.  This 
situation  led  to  an  undersirable  circumstance  where  operational  suitability 
could  not  be  determined  until  a  large  number  of  units  were  deployed. 

An  intimate  interaction  between  production  engineers  and  design 
engineers,  including  test  personnel,  should  be  established  early  in  the 
program.  This  process  will  tend  to  minimize  the  changes  between  the 
R&D  test  hardware  and  the  production  units;  hence,  there  is  less  pro¬ 
bability  that  new  problems  will  show  up  in  the  follow-on  T&E  tests  of  the 
produced  system.  Some  programs  have  done  this  splendidly  and  saved  much 
time  and  money  in  moving  from  development  to  production. 

5.  RUNNING  EVALUATION  OF  TESTS 

Assure  that  running  evaluations  of  tests  are  conducted.  If  it 
becomes  clear  that  test  objectives  are  unattainable  or  that  addi¬ 
tional  samples  will  not  change  the  test  outcome,  ensure  procedures 
are  established  for  terminating  the  test. 

In  contemporary  development  flight  testing,  aircraft-to-ground 
telemetry  and  computerized  data  reduction  provide  real-time  readout  of  test 
data,  and  make  a  running  evaluation  of  the  test  results  possible.  This 
permits  testers  to  identify  and  stop  unproductive  or  unnecessary  tests, 
or  to  modify  the  test  as  dictated  by  current  data. 

In  an  attack  aircraft  program,  a  considerable  number  of  rocket 
launching  missions  were  planned  to  demonstrate  accuracy.  After  these 
tests  were  completed  the  data  were  analyzed  and  showed  that  little  was 
learned  beyond  the  first  few  missions;  hence,  many  missions  were  wasted. 

6.  SIMULATION 

Analysis  and  simulation  should  be  conducted,  where  practicable,  before 
each  phase  of  development  flight  testing. 

Analysis,  simulation,  and  oth’r  ground  testing  should  be  used  to 
predict  test  outcome  and  to  establish  test  objectives.  The  flight  test 
may  then  be  accomplished  to  achieve  the  objectives.  Comparison  of 
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simulation  and  flight  test  results  provides  better  understanding  of  the 
system.  In  some  cases,  the  analysis  may  predict  dangerous  situations 
where  flight  testing  would  be  highly  risky.  In  one  instance,  the  con¬ 
tractor  pointed  this  out  and  discouraged  spin  testing.  The  service  decided 
to  conduct  the  spin  tests  and  lost  the  airplane. 

7.  SOFTWARE  TESTS 

Test  and  evaluation  should  ensure  that  software  products  are  tested 
appropriately  during  each  phase. 

Software  has  often  been  developed  more  as  an  add-on  than  an  inte¬ 
gral  part  of  the  overall  system.  Software  requirements  need  the  same  con¬ 
sideration  as  hardware  requirements  in  the  validation  phases.  Usual 
practices  often  do  not  sufficiently  provide  for  testing  the  software  sub¬ 
system  concept.  Often  the  facilities  available  to  contractors  for  soft¬ 
ware  development  and  verification  are  critical  to  schedule  and  cost. 

8.  AVIONICS  MOCK-UP 

Encourage  use  of  a  complete  avionics  system  installed  in  a  mock-up 
of  the  appropriate  section  or  sections  of  the  aircraft. 

The  avionics  mock-up  can  be  an  important  part  of  a  system  integration 
laboratory,  the  key  functions  of  which  are  subsystem  testing,  systems  in¬ 
tegration  testing  and  functional  compatibility  tests.  The  unit  can  be  used 
to  provide  early  crew  training  and  support  flight  tests. 

One  typical  fighter  program  used  the  avionics  mock-up  to  solve  air¬ 
plane  wiring  and  interface  problems,  to  minimize  airplane  down  time  for 
system  checkout,  to  provide  a  rapid  solution  to  flight  test  problems,  and 
to  verify  technical  manual  procedures. 

9.  ESCAPE  SYSTEM  TESTING 

Ensure  the  aircrew  escape  system  is  thoroughly  tested  with  particular 
atf  ition  to  redundant  features,  such  as  pyrotechnic  firing  channels. 
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Loss  of  aircrews  due  to  escape  system  faults  has  focused  wide-spread 
attention  to  this  system;  its  vital  operation  must  be  demonstrated  as 
highly  reliable  during  the  development  phase.  Redundant  systems  must  be 
instrumented  so  that  even  though  the  system  functions  satisfactorily  in 
test,  any  failure  of  a  redundant  circuit  will  be  detected.  The  failure 
can  then  be  analyzed  and  fixed,  reducing  the  possibility  that  redundant 
channels  will  fail  at  the  same  critical  time. 

Emergency  egress  from  a  first-line  fighter  aircraft  system  has 
been  only  77  perct?nt  successful.  A  detailed  investigation  of  the  escape 
system  disclosed  sufficient  failures  of  one  of  the  redundant  pyrotechnic 
channels  in  the  escape  actuation  system  so  that  failure  of  both  channels 
was  not  as  remote  a  possibility  as  thought. 

10.  STRUCTURAL  TESTING 

Assure  that  fatigue  testing  is  conducted  on  early  production  air¬ 
frames.  Airframe  production  should  be  held  to  a  low  rate  until 

satisfactory  progress  is  shown  in  these  tests. 

Airframe  fatigue  test  programs  have  identified  serious  structural 
problems.  Early  identification  of  unsatisfactory  conditions  has  resulted 
in  structural  fixes  or  modified  operations  procedures  so  that  unsafe  flight 
conditions  are  obviated. 

Fatigue  testing  should  be  accomplished  on  an  early  production  air¬ 
frame;  testing  a  prototype  or  a  limited  production  structural  model  may 
not  yield  data  representative  of  the  production  configuration.  Until  the 
structural  model  survives  critical  testing,  production  should  be  held  at 
a  low  rate  to  obviate  an  airframe  modification  program  involving  many 
delivered  aircraft. 

Major  failure  of  a  fatigue  test  article  before  test  objectives  have 
been  met  required  redesign  of  the  structure,  fabrication  of  the  newly 
designed  article,  and  reinitiation  of  fatigue  testing  on  the  article. 

During  the  structural  testing  of  an  airlift  aircraft,  fatigue  testing  was 
begun  on  the  critical  section  of  any  early  limited  production  model.  When 
it  failed  early,  a  model  of  the  refined  "production  configuration"  was 
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fatigue  tested  and  it,  too,  failed  sooner  than  desired.  Operational  limi¬ 
tations  were  imposed  on  the  aircraft  and  a  third  structure  representative 
of  the  "limited  use  aircraft"  was  fabricated  for  test.  The  fatigue  tests 
identified  problems  early;  load  limitations  were  placed  on  the  airplane 
and  fixes  initiated  so  that  it  could  be  operated  safely  in  peace  time 
while  maintaining  a  higher  potential  for  wartime  use. 

After  fatigue  test  objectives  have  been  met,  it  is  advantageous  to 
continue  testing  the  structural  model  until  massive  failure.  This  provides 
additional  knowledge  of  the  aircraft  in  test,  as  well  as  data  which  may  be 
applicable  to  future  design. 

11.  GUN  FIRING  TESTS 

All  forms  of  ordnance,  and  especially  those  which  create  gases  must 
be  fired  from  the  aircraft  for  external  effects  (blast  and  debris), 
internal  effects  (shock),  and  effects  on  the  propulsion  (inlet  com¬ 
position  or  distortion) . 

Ordnance  delivery  considerations  are  extremely  important  in  configur¬ 
ing  the  airplane.  Ordnance  tests  should  be  run  early  in  DT&E  to  ensure 
satisfactory  operation.  Tests  have  uncovered  serious  design  problems  re¬ 
sulting  from  the  ordnance  arrangement.  For  example,  one  interceptor  con¬ 
figuration  had  to  be  changed  because  the  gases  from  rockets  launched  from 
the  fuselage  stalled  the  engine.  The  redesign  placed  the  rockets  in  wing 
pods.  In  another  fighter,  gun  gas  and  link  and  cartridge  ejection  from 
a  fuselage-mounted  high  rate  of  fire  gun  caused  a  propulsion  problem.  On 
a  third  fighter,  debris  resulting  from  dropping  a  banded  bomb  cluster 
damaged  the  horizontal  tail  in  early  tests. 

12.  POST  STALL  CHARACTERISTICS 

Special  attention  is  warranted  un  the  post  stall  test  plans  for  DT&E 
and  OT&E. 

The  purpose  of  this  testing  is  to  determine  aircraft  controllability 
at  high  angles  of  attack,  in  stall  and  post-stall  conditions.  This  is  rel¬ 
atively  hazardous;  thus,  only  areas  of  the  performance  envelope  where  stalls 
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are  most  likely  to  occur  normally  need  to  be  investigated  in  flight.  For 
example,  fighter  aircraft,  designed  for  high-speed  maneuvering  flight, 
should  be  examined  for  high  speed  accelerated  stall  and  post-stall  char¬ 
acteristics.  This  type  of  testing  should  be  avoided  with  aircraft  designed 
for  low  "G"  operation.  Stall  and  post-stall  characteristics  of  all  types 
of  aircraft  should  be  checked  in  their  approach  and  landing  configurations. 

To  investigate  other  edges  of  the  performance  envelope  and  acquire 
data  otherwise  obtained  by  high  risk  to  aircraft  and  crew,  instrumented 
scale  models  can  be  dropped  from  aircraft,  or  spin  tunnels  and  computerized 
simulations  can  provide  other  methods  of  investigating  post  stall  character¬ 
istics. 

Contractor  demonstration  of  spin  recovery  is  usually  required  of  air¬ 
craft  most  likely  to  be  spun,  i.e.,  fighter,  attack,  and  some  trainer  air¬ 
craft.  Once  spins  are  accomplished,  duplication  should  not  be  attempted 
on  high  performance  aircraft  unless  there  is  strong  justification.  Two 
major  development  programs  in  the  early  70' s  lost  test  aircraft  in  the 
Services  post  stall  testing. 


13.  SPECIAL  ATTENTION  ITEMS 

Selected  hardware  items  sometimes  warrant  special  attention  by  test¬ 
ing  elements. 


There  are  a  number  of  components  and  parts  that  often  present  signifi¬ 
cant  recurring  problems  as  a  result  of  testing.  Some  of  these  are: 

(a)  Connectors  -  Connectors  are  a  source  of  trouble.  The  major  umbil¬ 
ical  connectors,  generally  used  as  the  electrical  interface  be¬ 
tween  ordnance  and  the  launcher  are  particularly  troublesome. 

It  is  advisable  to  do  everything  practicable  to  insure  the  con¬ 
tinuity  of  the  circuits  by  electrical  checks,  functional  checks, 
direct  viewing,  or,  in  some  cases,  x-ray  inspection.  Bent  pins, 
improper  mating,  and  intermittent  contacts  on  some  pins  have 
plagued  major  programs,  both  in  development  and  the  operational 
phases.  When  the  circuits  are  related  to  warheads  or  range 
safety  destruct  gear,  it  usually  is  not  practical  to  make  final 
electrical  continuity  checks  across  the  umbilical  connector. 
Connector  arrangements  whereby  the  mating  of  the  pins  for  these 
circuits  could  be  visually  inspected  would  be  desirable,  and  this 
aspect  should  he  stressed  by  T&E  agencies  at  design  review. 
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In  addition  to  the  test  content  for  electrical  continuity  across 
the  connectors,  a  logical  operational  suitability  test  would  be 
one  wherein  the  objective  would  be  to  show  proneness  of  failure 
of  a  particular  connector.  For  example,  how  easy  is  it  to  bend 
a  pin  so  that  the  connector  will  mate  but  the  bent  pin  will  be 
open?  Or,  can  adjacent  umbilical  connector  sets  be  mismated? 

(b)  Lanyards  -  Lanyards  and  other  safety  devices  whic>  preclude  in¬ 
advertent  launches  and  operations ,  can  also  sometimes  inadver¬ 
tently  preclude  a  successful  launch.  When  a  launch  is  commanded, 
the  software  should  be  scheduled  so  that  safety  pin  extraction 
action  operates  over  a  sufficient  time  period  so  that  variations 
can  be  accommodated  in  the  time  it  takes  the  stores  to  go  through 
the  launch  process. 

(c)  Safina.  Armine  and  Fuzing  -  Laboratory  devices  which  can  simu¬ 
late  the  environment  that  a  fuze  will  experience  during  its 
captive,  in-flight  and  impact  phases  should  be  liberally  used. 

It  is  expensive  and  difficult  to  operationally  test  fuzes  over 
the  range  of  environments  expected  and  often  much  of  the  testing 
can  be  done  separately  from  the  complete  weapon  system.  Alterna¬ 
tive  safing,  arming  and  fuzing  (SAF)  concepts  can  be  evaluated 
inexpensively,  with  ease  and  safety  through  the  use  of  such 
devices.  However,  finally  the  SAF  should  be  included  in  all  the 
full  system  tests  and  in  as  many  other  flight  tests  as  practical 
to  insure  that  subtle  effects  are  not  present  in  flight  tests. 


14.  SUB-SYSTEM  PERFORMANCE  HISTORY 

During  DT&E  and  IOT&E  of  aircraft,  ensure  a  performance  history  of 
each  sub-system  of  the  aircraft  will  be  kept. 

Development  is  an  iterative  process.  Many  pieces  of  equipment  are 
run  many  hours  in  perfecting  it  or  interconnected  gear.  When  equipment 
is  installed  in  the  aircraft  for  systems  tests,  it  is  particularly  desir¬ 
able  to  maintain  a  performance  history  on  each  subsystem.  Each  component 
of  the  system  should  be  identified  and  a  performance  history  kept  which 
allows  an  analysis  of  its  performance  with  respect  to  reliability,  main¬ 
tainability,  availability,  etc.  The  record  would  be  particularly  helpful 
in  analyzing  avionics  performance. 


15.  FLIGHT  DEFICIENCY  REPORTING 

Composition  of  flight  deficiencies  reported  by  aircrews,  particularly 
those  pertaining  to  avionics,  should  be  given  special  attention. 
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To  compose  meaningful  "write-ups"  (routine  aircrew  prepared  defici¬ 
ency  reports  after  a  flight)  test  aircrew  members  require  system  orienta¬ 
tion  and  training;  in  the  past  they  have  often  improperly  written-up  al¬ 
leged  malfunctions  because  they  did  not  understand  the  operations  or  the 
function  of  a  given  avionics  set.  Sometimes  the  unsatisfactory  operation 
resulted  from  the  wrong  switch  being  used.  Write-ups  caused  by  a  lack  of 
training  and  understanding  result  in  wasted  trouble  shooting.  Additional 
test  time  is  then  needed  to  repeat  the  test  with  the  proper  operator 
actions. 

Also  ensure  that  the  test  discrepancy  reporting  format  throughout 
DT&E  and  OT&E  phases  provides  a  consistent  basis  for  test  evaluation  during 
the  program.  It  is  important  to  clearly  identify  the  cause  of  failure 
and  to  determine  if  it  is  random.  This  can  best  be  accomplished  if  accu¬ 
rate  correlation  of  the  system  deficiency  history  is  possible. 

16.  CREW  LIMITATIONS 

Ensure  aircrew  limitations  are  included  in  the  tests. 

Most  air  launched  tactical  missiles,  especially  those  used  in  close 
support,  require  visual  acquisition  of  the  target  by  the  air  crew  and/or 
an  air/ground  controller.  Thus,  the  system  performance  is  very  much 
dependent  upon  visual  acquisition.  The  ability  to  acquire  the  target  must 
not  be  assumed  trivial  and  must  be  tested  in  operationally  realistic  con¬ 
ditions  to  determine  to  what  extent  this  factor  may  limit  the  capability 
of  the  total  system. 

In  a  normal  launch  sequence  of  an  electro-optical  (E-0)  type  missile, 
the  crew  is  required  to  (a)  locate  and  acquire  the  target  visually,  (b) 
determine  that  what  he  sees  is  a  real  target,  (c)  decide  whether  it  lies 
within  the  size,  contrast,  and  rarge  envelope  of  the  weapon,  (d)  re-acquire 
the  target  on  his  cockpit  display  by  means  of  the  weapon's  seeker,  (e) 
verify  that  the  seeker  has  "locked-on"  the  proper  target,  (f)  launch  the 
weapon,  and  (g)  maneuver  away  from  the  target  area.  This  sequence  of 
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activities  must  be  conducted  in  a  dynamic  (and  very  likely  hostile)  envir¬ 
onment;  the  effect  is  to  make  system  performance  very  much  dependent  upon 
the  operator's  ability. 

17.  USE  OF  OPERATIONAL  PERSONNEL 

Recommend  experienced  operational  personnel  help  in  establishing 
measures  of  effectiveness  and  in  other  operational  test  planning. 

In  conducting  OT&E,  use  typical  operational  aircrews  and  support 
personnel . 

Experienced  operators  should  participate  in  developing  measures  of 
effectiveness  and  in  planning  the  operational  test  so  that  results  will  be 
operationally  significant.  There  was  a  tendency  in  the  past  for  the 
initial  operational  command  tests  to  be  largely  a  repeat  or  extension  of 
the  DT&E  activities.  The  result  was  that  these  tests  were  of  little 
significance  in  determining  operational  adequacy. 

Economical  utilization  of  resources  suggests  that  personnel  gaining 
experience  in  planning  and  conducting  IOT&E  on  an  aircraft  system  should 
be  utilized  to  plan  the  follow-on  OT&E. 

Typical  operational  unit  personnel  should  conduct  the  operational 
test.  The  use  of  lead  crews  and  highly  experienced  maintenance  crews  and 
specialists  may  bias  the  outcome  of  the  evaluation  by  providing  informa¬ 
tion  which  may  later  prove  to  be  optimistic.  For  example,  an  aircrew  which 
carries  a  lead  navigator  generally  delivers  weapons  much  more  accurately 
than  a  recently-designated  combat-ready  aircrew  with  a  relatively  inex¬ 
perienced  navigator.  Highly  qualified  operational  evaluators  should  parti¬ 
cipate  in  planning  operational  tests,  in  evaluating  results  and  in  some 
cases  in  conducting  the  tests  where  the  bias  is  understood. 

18.  ROLE  OF  THE  USER 

Ensure  that  users  participate  in  the  T&E  phases  so  that  their  needs 
are  represented  in  the  development  of  the  system  concept  and  hard- 
ware. 
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Initially  the  operational  command  should  play  an  advisory  role  during 
feasibility  and  development  testing,  and  should  play  a  more  significant 
role  in  the  conduct  of  the  test  program  as  it  becomes  operational. 

In  a  case  where  the  user  was  not  active  during  the  development  of 
a  tactical  fighter  system,  the  scope  displays  of  the  system  were  not  fully 
satisfactory  to  the  operational  command. 

19.  CREW  FATIGUE  AND  SYSTEM  EFFECTIVENESS 

In  attack  aircraft  operational  testing,  and  particularly  in  attack 
helicopter  tests  where  vibration  i?  a  fatiguing  factor,  ascertain 
that  the  tests  include  a  measure  cf  degradation  over  time. 

Short,  few-per-day  flights  may  not  simulate  the  operational  situa¬ 
tion  where  many  sorties  in  a  hostile  environment  would  be  expected  to  take 
place  at  a  high  rate  per  day  over  a  long  period.  Within  the  bounds  of 
safety  to  the  crews,  they  should  perform  in  situations  where  data  can  be 
collected  relative  to  weapon  system  effectiveness  as  a  function  of  human 
fatigue. 

20.  TEST  CONSTRAINTS  ON  CREW 

Detailed  operational  test  plans  should  be  evaluated  to  determine  that 
the  test  imposed  conditions  on  the  crew  do  not  invalidate  the  appli¬ 
cability  of  the  data  so  collected. 

The  evaluation  of  realistic  performance  must  be  a  major  objective 
in  any  OT&E.  Constraints  such  as  requiring  the  crew  to  perform  atypical 
functions,  or  unrealistic  test  conditions  may  throw  doubt  on  the  validity 
of  the  data  as  it  relates  to  weapon  system  effectiveness.  Realistic 
combat  operator  task  loading  should  be  included  to  the  extent  possible. 

One  fighter  program  carried  on  OT&E  in  phases.  Avionics  testing 
was  stressed  during  one  period.  Considerable  care  must  be  taken 
to  insure  that  operational  testing  of  this  nature  yields  realistic 
results.  For  example,  on  this  same  fighter,  during  training  and  probably 
during  the  avionics  operational  test  phase,  the  second  crew  man  helped  the 
pilot  considerably  by  monitoring  radar  and  other  displays  during  low  level 
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penetration;  however,  in  combat  penetrations  it  was  found  that  the  pilot 
got  much  less  help  during  this  critical  phase  because  the  number  two  man 
was  involved  in  electronic  warfare  matters. 

21.  COMPLETE  BASIC  DT&E  BEFORE  STARTING  IOT&E 

Before  a  weapon  system  is  subjected  to  IOT&E,  all  critical  subsystems 

should  have  completed  basic  DT&E  with  the  significant  problems  solved. 

Occasionally  development  deficiencies  have  been  carried  into  IOT&E. 
This  is  undesirable  as  it  impedes  meaningful  operational  testing.  Solu¬ 
tions  to  unsolved  development  problems  are  difficult  to  obtain  under  rel¬ 
atively  austere  field  conditions  of  IOT&E  away  from  laboratory  personnel 
and  their  diagnostic  equipment.  Technical  problems  that  surface  in  IOT&E 
can  sometimes  be  most  efficiently  addressed  by  the  contractor's  use  of  an 
instrumented  development  test  aircraft. 

On  a  new  fighter  program  development  testing  of  digital  avionics  was 
carried  over  into  initial  operational  testing.  Test  results  provided 
little  significant  operational  information. 

22.  REALISM  IN  TESTING 

Ascertain  that  final  DT&E  system  tests  and  IOT&E  flight  tests  are 

representative  of  operational  conditions. 

An  attack  airplane  successfully  completed  DT&E  and  OT&E  type  testing, 
but  structural  problems  showed  up  in  combat  operations.  All  the  testing 
was  done  from  bases  with  good,  smooth  ramps,  taxiways  and  runways.  The 
combat  airfields  had  much  poorer  ramps  and  taxiways  which  resulted  in 
structural  problems  for  the  new  airplane. 

Operational  testing  must  be  planned  and  conducted  as  realistically  as 
practicable  so  as  to  preclude  operational  deficiencies  in  combat  and  so  as 
to  minimize  costly  modifications  of  production  aircraft. 
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23.  TEST  ALL  PLANNED  PROFILES  AND  MODES 


Tests  should  be  conducted  to  evaluate  all  planned  operational  flight 

profiles  and  all  primary  and  back-up,  degraded  operating  modes. 

The  flight  profiles  need  to  be  evaluated  to  ensure  that  no  unpredicted 
problems  occur.  In  aircraft  test  programs  the  primary  design  profiles  are 
generally  well  tested.  The  OT&E  efforts  re-test  these  design  requirements 
to  insure  they  can  be  satisfied  in  operational  use.  The  testing  should 
also  cover  those  operational  profiles  which  stress  the  system.  In  addi¬ 
tion,  the  operational  tests  should  exercise  backup  operating  modes. 

The  backup  operating  modes  need  to  be  tested  because  aircraft 
avionics  reliability  problems  and  battle  damage  may  cause  the  primary 
mode  to  fail.  For  instance,  a  carrier  may  have  several  navigation  systems 
such  as  inertial,  doppler,  or  bomb/nav  radar  to  provide  position  data  for 
air  launched  missile  inputs.  All  systems  might  be  used  in  the  primary 
operating  mode.  In  an  operational  environment,  it  is  not  unlikely  that 
some  launches  might  be  made  with  only  one  of  these  systems  remaining  in 
operation.  Therefore,  these  backup,  degraded  operating  modes  should  be 
tested  to  determine  their  characteristics  and  capabilities. 

24.  UPDATE  OF  OPERATIONAL  TEST  PLANS 

Ensure  operational  test  plans  are  reviewed  and  updated  as  needed 

to  make  them  relevant  to  evolving  concepts. 

Operational  concepts  on  which  aircraft  design  is  predicated  change  as 
time  passes.  It  is  essential  that  operational  test  plans  be  based  on  the 
latest  version  of  the  operational  concept.  In  one  program  If  the  helicop¬ 
ter  had  been  tested  on  the  basis  of  an  OT&E  plan  prepared  at  initiation  of 
the  development  contract,  satisfactory  results  would  not  have  proven  any¬ 
thing  because  of  changes  in  the  concept  of  employment,  environment,  and 
tactics. 

In  another  case,  valid  operational  information  relative  to  an  airlift 
aircraft  limited  by  structural  problems  was  obtained  from  OT&E  because  the 
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test  plan  was  modified  to  accommodate  major  changes  to  the  original  opera¬ 
tional  concept. 

25.  CONDUCT  IOT&E  EARLY 

Ensure  operational  suitability  tests  are  planned  to  attempt  to  iden¬ 
tify  operational  deficiencies  of  new  systems  quickly  so  that  fixes 
can  be  developed  and  tested  before  large  scale  production. 

Continually  update  OT&E  plans  so  that  testing  can  be  expedited  as 
soon  as  it  is  appropriate  to  start.  Look  at  suspected  problem  areas  early. 
For  instance,  operational  tests  should  be  used  to  identify  problem  areas 
associated  with  the  tactics  of  delivering  a  missile  in  realistic  combat 
conditions  which  may  significantly  degrade  the  operational  effectiveness 
or  utility  of  the  weapon  system.  For  example,  an  optically  guided  mis¬ 
sile  tracking  subsystem  may  not  function  as  desired  for  some  sun  angle 
conditions,  or  a  semi-active  laser  seeker  guided  missile  requires  reflect¬ 
ed  energy  from  the  target,  thus  imposing  possible  constraints  on  the  geo¬ 
metry  of  the  designator,  the  launch  airplane  and  the  target.  Human 
factor  problems  associated  with  displays,  task  loading  and  weapons  deliv¬ 
ery  functions  may  restrict  total  capability  of  the  system. 

These  problem  areas  need  to  be  identified  early  in  the  IOT&E  phase 
so  that  possible  engineering  fixes  can  be  studied,  developed  and  tested 
prior  to  large  scale  production  acquisition  to  allow  for  a  more  tactic¬ 
ally  usable  missile.  Problem  areas  associated  with  human  factors  can  be 
examined  to  determine  special  training  requirements  or  hardware  and  soft¬ 
ware  modifications. 

26.  MISSILE  LAUNCH  TESTS 

Review  the  final  position  fix  planned  before  launching  inertial  guided 
air-to-surface  missiles. 

In  operational  tests  of  bomber/missile  systems,  there  have  been  in¬ 
stances  in  which  artificial  position  fixes  were  utilized  just  prior  to 
missile  launch.  Optimistic  missile  accuracy  data  results,  unless  the  posi¬ 
tion  fixing  test  procedures  are  similar  to  those  used  for  operational 
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missiles.  System  accuracy  for  inertial  guided  air-launched  missiles  will 
be  most  dependent  upon  position,  velocity,  and  heading  data  provided  by 
the  bomber.  If  the  test  program  uses  beacons,  radar  reflectors  or  other 
cooperative  accurate  fix  features  in  the  near  vicinity  of  the  target,  the 

accuracy  results  will  be  better  than  could  be  expected  on  an  operational 

mission.  An  operational  bomber  may  have  to  plan  to  make  the  last  fix  on 
a  prominent  feature  at  a  considerable  distance  from  the  target  due  to  the 
possibility  of  battle  damage  or  countermeasures  near  the  target. 

27.  OPERATIONAL  TEST  REALISM 

Ensure  operational  testing  is  realistic. 

For  example,  in  testing  the  operational  navigational  capability  of  an 
aircraft  to  reach  a  briefed  target,  the  up-dating  procedures  used  during 
the  test  should  be  those  expected  in  actual  employment.  If  the  navigation 
system  is  up-dated  in  an  unrealistic  fashion,  the  operational  effective¬ 
ness  of  the  system  in  terms  of  the  ability  of  the  aircraft  to  acquire  the 

enemy  target  may  be  biased  towards  a  best  possible  condition.  A  recent 
operational  test  was  conducted  where  the  aircrew  was  advised  of  the  amount 
of  their  navigation  error  and  were  asked  to  correct  so  that  a  target  could 
be  acquired.  This  compromised  operational  realism. 

28.  MISSION  COMPLETION  SUCCESS  PROBABILITY 

Mission  completion  success  probability  factors  should  be  used  to 

measure  progress  in  the  aircraft  test  program. 

The  test  program  should  use  mission  completion  success  probability 
(MCSP)  as  a  management  tool  to  assess  on  a  continuing  basis  the  capability 
of  the  system  to  meet  operational  commitments.  Standardized  measures  which 
evaluate  overall  system  performance  and  are  established  as  system  develop¬ 
ment  goals  are  more  significant  than  non-standardized  component  bench  test 
specifications  and  individual  subsystem  reliability  predictions.  For 
example,  avionics  MTBFs  on  bench  qualification  tests  usually  run  some  large 
factor  better  than  the  MTBF  of  the  subsystem  when  it  is  installed  in  the 
aircraft.  Aircraft  peculiar  environmental  factors  like  actual  temperatures 
experienced,  vibration,  and  G's  are  difficult  to  adequately  simulate  in 
bench  testing  or  system  integration  laboratory  tests. 
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IV.  SUBSTANTIAL  PRODUCTION/DEPLOYMENT  PHASE 


This  phase  occurs  after  the  DSARC  substantial  production  decision. 
Follow-on  OT&E  will  be  conducted  with  production  hardware.  The  test  and 
evaluation  checklist  includes: 

1.  Operational  Test  Realism 

2.  Design  OT&E  for  Less  than  Optimal  Conditions 

3.  New  Threat 

A.  Certification  of  Ordnance 

5.  Test  Fixes 

6.  Inadvertent  Influence  of  Test 

7.  Deficiencies  Discovered  In-Service 

8.  Lead  the  Fleet 

9.  Direct  Support  Aircraft 

The  reader  should  also  review  the  checklist  items  in  the  previous 
phases,  especially  the  last  phase,  since  many  of  these  items  will  be 
applicable  during  this  phase. 
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1. 


OPERATIONAL  TEST  REALISM 


Ascertain  operational  testing  Is  conducted  under  realistic  combat 

conditions. 

The  offense/defense  battle  needs  to  be  simulated  before  the  evalua¬ 
tion  of  the  weapon  system  can  be  considered  completed.  Whether  this 
exercise  is  conducted  within  a  single  service  (as  in  the  test  of  an  attack 
helicopter  against  tanks)  or  between  services  (as  in  the  test  of  an  attack 
aircraft  system  against  tanks  with  anti-aircraft  protection),  the  plans 
for  such  testing  should  be  formulated  as  part  of  the  system  development 
plan. 

Special  attention  should  be  placed  on  the  realistic  environment  to 
be  used  in  the  operational  test,  on  the  vulnerability  of  the  various  ele¬ 
ments  of  the  system  (FAC,  platform,  target  designator,  etc.)  and  on  the 
use  of  models  and  simulators  to  evaluate  the  post  launch  activities  affect¬ 
ing  target  kills  (probability  of  break  lock  of  missiles,  reliability, 
invalid  launches,  etc.)  when  the  ordnance  is  not  launched  during  tests. 

Similarly,  the  OT&E  offense/defense  tests  should  be  performed  in 
appropriate  weather.  The  practicality  of  finding  the  right  conditions 
for  some  of  these  tests  during  a  reasonable  IOT&E  may  require  some  condi¬ 
tions  to  be  simulated  and  possibly  some  to  be  deferred  as  goals  for  the 
follow-on  OT&E.  In  summary,  the  goal  is  that  the  full  production  equip¬ 
ment  should  be  tested  under  appropriate  offense/defense,  day/night, 
weather  conditions. 

2.  DESIGN  OT&E  FOR  LESS  THAN  OPTIMAL  CONDITIONS 

Structure  the  OT&E  logistical  support  for  simulated  combat  conditions. 

In  operationally  testing  tactical  fighter  aircraft,  the  aircraft 
should  be  operated  from  minimal  basing  facilities — relatively  unimproved 
runways,  taxiways,  and  ramps,  mobile  maintenance  shelters,  and  substandard 
administrative  and  living  accommodations.  Planning  for  support  of  the 
aircraft  under  these  conditions  should  be  exercised. 
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Although  these  conditions  may  seem  over demanding,  bare  base  conditions 
should  be  anticipated. 

3.  NEW  THREAT 

Be  alert  to  the  need  to  extend  the  OT&E  if  a  new  threat  shows  up. 

It  is  important  to  ensure  that  adequate  testing  of  the  aircraft  system 
is  carried  out  against  current  threats  and  that  the  T&E  be  responsive  tr 
new  threats,  should  they  be  identified  later. 

The  operational  test  should  be  re-instituted  to  evaluate  system  ade¬ 
quacy  in  the  face  of  new  threats  encountered  after  the  system  is  operational 
and  the  first  follow-on  OT&E  phase  is  complete.  The  continuing  training 
activities  that  usually  generate  reliability  an'*  CEP  data  for  war  plan  pur¬ 
poses,  should  be  considered  also  as  an  additional  means  for  testing  responses 
to  new  threats. 

4.  CERTIFICATION  OF  ORDNANCE 

Assure  that  ordnance  to  be  delivered  by  an  aircraft  system  is 

certified  for  that  aircraft. 

Most  military  aircraft  systems  are  flexible  enough  to  carry  a  consid¬ 
erable  spread  of  ordnance.  Sometimes  the  arrangements  to  qualify  the 
many  ordnance  options  on  the  aircraft  are  loosely  planned;  the  contractor 
does  some  of  the  work  and  the  Service  does  part.  In  addition,  there  are 
new  munitions  being  developed,  some  which  become  available  may  not  have 
been  included  in  the  initial  test  plans;  hence,  certification  of  ordnance 
may  be  required  during  the  production  and  deployment  phase. 

5.  TEST  FIXES 

Test  fixes  resulting  from  earlier  operational  testing. 

Following  initial  operational  tests  which  identify  problem  areas, 
follow-on  OT&E  should  investigate  the  adequacy  of  the  fixes  incorporated, 
particularly  if  the  IOT&E  did  not  run  long  enough  to  test  the  fixes. 
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For  example,  on  one  bomber  missile  program  a  rocket  nozzle  deficiency 
showed  up  in  a  ground  qualification  test  of  a  motor  just  at  the  end  of  the 
equivalent  of  IOT&E  testing.  Fixes  were  developed  and  ground  tested,  but 
extensive  full  system  flight  tests  with  the  fixed  nozzle  had  to  await 
follow-on  OT&E . 

6.  INADVERTENT  INFLUENCE  OF  TEST 

OT&E  plans  should  provide  measures  for  insuring  that  actions  by  obser¬ 
vers  and  umpires  do  not  unwittingly  influence  trial  outcomes. 

Observers  and  umpires  can  provide  important  clues  to  the  participants 
of  operational  suitability  offense/defense  testing,  and  in  that  way,  may 
lessen  the  validity  of  the  test.  For  example,  in  a  recent  test  program 
where  air/ground  duels  were  laing  conducted,  observers  looked  in  the 
direction  from  which  they  had  been  told  that  the  aircraft  would  approach. 
This  inadvertently  disclosed  the  direction  of  approach  to  the  ground  party 
in  the  duel.  Similarly,  concentrations  of  observers  at  a  certain  location 
may  clue  the  air  crews  where  to  search  first  for  the  ground  targets. 

Plans  should  obviate  compromising  the  test  in  this  manner  by  prescrib¬ 
ing  necessary  observer  and  umpire  discipline. 

7.  DEFICIENCIES  DISCOVERED  IN-SERVICE 

Be  aware  that  in-service  operations  of  an  aircraft  system  will  surface 

deficiencies  which  extensive  follow-on  OT&E  probably  would  not  uncover. 

Formally  conducted  OT&E  is  expensive,  time-consuming,  and  should  be 
stopped  when  the  basic  test  objectives  have  been  met.  In-service  operation 
of  an  aircraft  system  provides  rapid  accrual  of  flight  and  maintenance 
experience.  Routine  operations  and  training  will  serve  to  identify 
design  and  operational  deficiencies  not  found  in  OT&E.  Operational  organi¬ 
zations  often  have  maintenance  data  analysis  units,  and  they  recognize 
a  significant  problem  from  periodic  maintenance  inspections  or  from  aircrew 
reported  discrepancies.  Serious  problems  may  require  diagnostic  testing, 
development  of  a  fix,  and  even  additional  OT&E. 
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After  many  years  in  the  inventory,  a  strategic  bomber  system  experienc¬ 
ed  major  structural  fatigue  problems.  This  necessitated  a  major  modifica¬ 
tion  program  preceded  by  structural  testing  and  development  of  the  fix.  A 
good  fatigue  test  program  during  the  R&D  might  have  identified  this  problem 
before  all  the  aircraft  were  produced.  On  the  other  hand,  there  are  some 
items  which  can  only  be  economically  tested  in-service.  For  example,  it 
may  be  very  costly  to  attempt  to  test  all  combinations  and  permutations  of 
a  complex  avionics  digital  program  in  OT&E.  It  may  be  more  reasonable  to 
plan  for  in-service  correction  of  a  reasonable  number  of  things  that  can 
only  be  adequately  tested  in  the  fleet  environment  over  a  relatively  long 
period  of  use.  Contractors  may  be  required  to  provide  in-service  correc¬ 
tion  of  some  things  where  the  cost  of  specialist  support  in  the  field  is 
economical  compared  to  the  extensive  DT&E  and  OT&E  otherwise  required. 

8.  LEAD  THE  FLEET 

Accelerated  service  test  of  a  small  quantity  of  early  production 
aircraft  is  advisable  during  follow-on  OT&E  and  thereafter. 

These  accelerated  test  aircraft  which  are  flown  to  accumulate  flying 
hours  much  faster  than  normal  may  disclose  major  airframe  and  engine  de¬ 
ficiencies  which  the  entire  fleet  might  be  expected  to  experience  should 
early  corrective  action  not  be  taken.  Early  discovery  of  any  service  in¬ 
duced  deficiency  provides  more  time  to  develop  a  fix,  test  it,  and  start 
modification  of  the  fleet. 

A  trainer  aircraft  used  in  combat  as  a  forward  air  control  aircraft 
and  as  a  fighter-bomber  was  subjected  to  early  accelerated  service  test. 

A  major  propulsion  problem  was  surfaced  and  fixed  before  large  quantities 
of  these  aircraft  were  delivered. 

9.  DIRECT  SUPPORT  AIRCRAFT 

OT&E  for  close  support  fixed-wing  aircraft  and  fire  suppression  heli¬ 
copter  systems  should  emphasize  system  ordnance  delivery  accuracy  and 
system  survivability. 
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Where  an  aircraft  has  been  designed  for  close  air  support  under  low 
ceilings  and  poor  visibility,  it  should  be  tested  in  OT&E  under  these  con¬ 
ditions  with  live  ordnance  to  evaluate  its  unique  capability.  In  the  close 
air  support  role,  accuracy  is  of  primary  importance — accuracy  in  the  sense 
of  a  tight  cluster  of  bomb  impacts  around  the  target.  Accuracy  should  be 
determined  as  precisely  as  possible  with  particular  attention  to  the  gross 
errors,  as  ground  commanders  are  wary  of  air-dropped  "short  rounds" — 
inadvertent  delivery  of  ordnance  by  friendly  aircraft  on  friendly  ground 
troops. 

The  determination  of  survivability  should  also  receive  high  interest. 
Offense/defense  tests  should  be  conducted  to  measure  significant  parameters 
related  to  vulnerability,  such  as  the  following: 

•  Exposure  time  as  the  aircraft  penetrates  hostile  defenses,  maneuvers 
for  ordnance  delivery,  and  withdraws. 

•  Ranges  and  angles  from  the  defense  while  the  aircraft  is  observed. 

•  Sun  angle  as  it  relates  to  detectability  (glint  off  the  aircraft). 

•  Background  when  the  aircraft  pops  up  to  acquire  the  target  (e.g., 
sky  versus  terrain) . 

•  Aircraft  noise  as  a  function  of  wind  direction,  weather,  clouds, 
and  battlefield  noise. 

These  data  can  be  used  in  estimating  the  probability  of  survival  for 
the  aircraft. 
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